[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:51:40 dist-test-slave-gmf2 krb5kdc[20970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:51:40 dist-test-slave-gmf2 krb5kdc[20970](info): set up 2 sockets
Jun 21 06:51:40 dist-test-slave-gmf2 krb5kdc[20970](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260621 06:51:41.738754 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.009s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:51:41 dist-test-slave-gmf2 krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:51:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:51:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:51:41.907218 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:45661
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:45661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:42.203543 20983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:42.203856 20983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:42.204125 20983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:42.211238 20983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:51:42.211402 20983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:42.211529 20983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:42.211668 20983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:51:42.211750 20983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:51:42.221192 20983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:45661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:45661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:42.222465 20983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:42.223759 20983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:42.235257 20988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:42.235301 20989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:42.235323 20991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:42.235949 20983 server_base.cc:1034] running on GCE node
I20260621 06:51:42.236706 20983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:42.238090 20983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:42.239310 20983 hybrid_clock.cc:648] HybridClock initialized: now 1782024702239248 us; error 67 us; skew 500 ppm
I20260621 06:51:42.239637 20983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:42 dist-test-slave-gmf2 krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024702, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:42.242633 20983 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:51:42.244046 20983 webserver.cc:460] Webserver started at http://127.20.120.254:42731/ using document root <none> and password file <none>
I20260621 06:51:42.244730 20983 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:42.244874 20983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:42.245172 20983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:42.248168 20983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "9848d15507af4eb2a5d20451ba9a11df"
format_stamp: "Formatted at 2026-06-21 06:51:42 on dist-test-slave-gmf2"
server_key: "c61361151316f0c6e3b4fd44354e4bef"
server_key_iv: "e2123deadd96794af8c010ea52a0eca0"
server_key_version: "encryptionkey@0"
I20260621 06:51:42.248937 20983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "9848d15507af4eb2a5d20451ba9a11df"
format_stamp: "Formatted at 2026-06-21 06:51:42 on dist-test-slave-gmf2"
server_key: "c61361151316f0c6e3b4fd44354e4bef"
server_key_iv: "e2123deadd96794af8c010ea52a0eca0"
server_key_version: "encryptionkey@0"
I20260621 06:51:42.254038 20983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 06:51:42.257230 20998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:42.258528 20983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:51:42.258704 20983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "9848d15507af4eb2a5d20451ba9a11df"
format_stamp: "Formatted at 2026-06-21 06:51:42 on dist-test-slave-gmf2"
server_key: "c61361151316f0c6e3b4fd44354e4bef"
server_key_iv: "e2123deadd96794af8c010ea52a0eca0"
server_key_version: "encryptionkey@0"
I20260621 06:51:42.258888 20983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:42.273291 20983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:42.276927 20983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:42.277190 20983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:42.294440 20983 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:45661
I20260621 06:51:42.294461 21049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:45661 every 8 connection(s)
I20260621 06:51:42.295710 20983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:51:42.299140 21050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:42.301860 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 20983
I20260621 06:51:42.302201 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:51:42.304579 20963 external_mini_cluster.cc:1428] Setting key ec394b3f393cdaecc99ed76e1f6461c5
I20260621 06:51:42.311388 21050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df: Bootstrap starting.
I20260621 06:51:42.314241 21050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:42.315163 21050 log.cc:824] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:42.317699 21050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df: No bootstrap required, opened a new log
I20260621 06:51:42.327117 21050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9848d15507af4eb2a5d20451ba9a11df" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45661 } }
I20260621 06:51:42.327505 21050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:42.327611 21050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9848d15507af4eb2a5d20451ba9a11df, State: Initialized, Role: FOLLOWER
I20260621 06:51:42.328171 21050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9848d15507af4eb2a5d20451ba9a11df" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45661 } }
I20260621 06:51:42.328397 21050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:51:42.328541 21050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:51:42.328693 21050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:42.330811 21050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9848d15507af4eb2a5d20451ba9a11df" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45661 } }
I20260621 06:51:42.331235 21050 leader_election.cc:302] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9848d15507af4eb2a5d20451ba9a11df; no voters:
I20260621 06:51:42.331822 21050 leader_election.cc:288] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:51:42.331930 21055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [term 1 FOLLOWER]: Leader election won for term 1
Jun 21 06:51:42 dist-test-slave-gmf2 krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:42.333207 21055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [term 1 LEADER]: Becoming Leader. State: Replica: 9848d15507af4eb2a5d20451ba9a11df, State: Running, Role: LEADER
I20260621 06:51:42.333739 21055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9848d15507af4eb2a5d20451ba9a11df" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45661 } }
I20260621 06:51:42.334161 21050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:51:42.336421 21056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9848d15507af4eb2a5d20451ba9a11df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9848d15507af4eb2a5d20451ba9a11df" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45661 } } }
I20260621 06:51:42.336414 21057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9848d15507af4eb2a5d20451ba9a11df. Latest consensus state: current_term: 1 leader_uuid: "9848d15507af4eb2a5d20451ba9a11df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9848d15507af4eb2a5d20451ba9a11df" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45661 } } }
I20260621 06:51:42.336836 21056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [sys.catalog]: This master's current role is: LEADER
I20260621 06:51:42.336839 21057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df [sys.catalog]: This master's current role is: LEADER
I20260621 06:51:42.347296 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:42.308889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.20.120.254:45661)
0621 06:51:42.309763 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:42.309777 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:42.311057 (+ 1280us) server_negotiation.cc:407] Connection header received
0621 06:51:42.319890 (+ 8833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:42.319913 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:42.320372 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:42.320767 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:42.323295 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:42.325285 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:42.326958 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:42.327880 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:42.328073 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:42.334173 (+ 6100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:42.334185 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:42.334202 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:42.334244 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:42.339057 (+ 4813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:42.340330 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:42.340341 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:42.340351 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:42.340600 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:42.341242 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:42.341250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:42.341255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:42.341968 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:42.344881 (+ 2913us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:42.345204 (+ 323us) server_negotiation.cc:299] Negotiation successful
0621 06:51:42.345698 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":433,"threads_started":1}
I20260621 06:51:42.351116 21060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:51:42.356089 21060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:51:42.363879 21060 catalog_manager.cc:1347] Generated new cluster ID: 7b06f45d78fa4f67a8b85fa83dcb97cd
I20260621 06:51:42.363976 21060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:51:42.383770 21060 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:51:42.385532 21060 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:51:42.392269 21060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9848d15507af4eb2a5d20451ba9a11df: Generated new TSK 0
I20260621 06:51:42.393292 21060 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:51:42.435675 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45661
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:42.737802 21076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:42.738102 21076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:42.738307 21076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:42.744968 21076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:42.745077 21076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:42.745275 21076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:51:42.754347 21076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45661
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:42.755471 21076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:42.756680 21076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:42.769008 21081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:42.769034 21082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:42.769073 21084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:42.769613 21076 server_base.cc:1034] running on GCE node
I20260621 06:51:42.770165 21076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:42.770913 21076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:42.772119 21076 hybrid_clock.cc:648] HybridClock initialized: now 1782024702772069 us; error 47 us; skew 500 ppm
I20260621 06:51:42.772356 21076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:42 dist-test-slave-gmf2 krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024702, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:42.779066 21076 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:51:42.780366 21076 webserver.cc:460] Webserver started at http://127.20.120.193:46421/ using document root <none> and password file <none>
I20260621 06:51:42.781015 21076 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:42.781159 21076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:42.781450 21076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:42.784152 21076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "90f7693c1df54bac9bd2b7326db1a3a5"
format_stamp: "Formatted at 2026-06-21 06:51:42 on dist-test-slave-gmf2"
server_key: "b0d7400ba9ca802c2b1c7c72e4df85f5"
server_key_iv: "b530acb9864a5616a23f105dd5f26a0e"
server_key_version: "encryptionkey@0"
I20260621 06:51:42.784849 21076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "90f7693c1df54bac9bd2b7326db1a3a5"
format_stamp: "Formatted at 2026-06-21 06:51:42 on dist-test-slave-gmf2"
server_key: "b0d7400ba9ca802c2b1c7c72e4df85f5"
server_key_iv: "b530acb9864a5616a23f105dd5f26a0e"
server_key_version: "encryptionkey@0"
I20260621 06:51:42.789696 21076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260621 06:51:42.792961 21091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:42.794258 21076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:51:42.794417 21076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "90f7693c1df54bac9bd2b7326db1a3a5"
format_stamp: "Formatted at 2026-06-21 06:51:42 on dist-test-slave-gmf2"
server_key: "b0d7400ba9ca802c2b1c7c72e4df85f5"
server_key_iv: "b530acb9864a5616a23f105dd5f26a0e"
server_key_version: "encryptionkey@0"
I20260621 06:51:42.794602 21076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:42.806129 21076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:42.809480 21076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:42.809742 21076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:42.810597 21076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:51:42.812498 21076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:51:42.812592 21076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:42.812705 21076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:51:42.812785 21076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:42.843012 21076 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:43167
I20260621 06:51:42.843056 21203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:43167 every 8 connection(s)
I20260621 06:51:42.844228 21076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:51:42.850368 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 21076
I20260621 06:51:42.850615 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:51:42.851366 20963 external_mini_cluster.cc:1428] Setting key 9afd6a2183e0aa0601365658cef5afdf
Jun 21 06:51:42 dist-test-slave-gmf2 krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024702, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:42.865587 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:42.846914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:57525 (local address 127.20.120.254:45661)
0621 06:51:42.847090 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:42.847095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:42.848165 (+ 1070us) server_negotiation.cc:407] Connection header received
0621 06:51:42.850424 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:42.850429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:42.850525 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:42.850626 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:42.852838 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:42.853745 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:42.855289 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:42.855669 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:42.855797 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:42.859547 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:42.859552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:42.859556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:42.859587 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:42.862384 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:42.863120 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:42.863126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:42.863129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:42.863331 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:42.863776 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:42.863782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:42.863786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:42.864095 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:42.864254 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:42.864973 (+ 719us) server_negotiation.cc:299] Negotiation successful
0621 06:51:42.865198 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
I20260621 06:51:42.866808 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:42.847356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45661 (local address 127.20.120.193:57525)
0621 06:51:42.848021 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:42.848056 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:42.850167 (+ 2111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:42.851062 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:42.851072 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:42.851937 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:42.852668 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:42.852685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:42.853886 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:42.853892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:42.855148 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:42.855159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:42.855886 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:42.855893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:42.856136 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:42.856786 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:42.856807 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:42.859328 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:42.862529 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:42.862538 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:42.862553 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:42.862989 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:42.863443 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:42.863451 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:42.863456 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:42.863647 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:42.864226 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:42.864234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:42.864592 (+ 358us) client_negotiation.cc:769] Sending connection context
0621 06:51:42.864864 (+ 272us) client_negotiation.cc:240] Negotiation successful
0621 06:51:42.865348 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":265,"threads_started":1}
I20260621 06:51:42.868872 21204 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45661
I20260621 06:51:42.869146 21204 heartbeater.cc:464] Registering TS with master...
I20260621 06:51:42.869810 21204 heartbeater.cc:511] Master 127.20.120.254:45661 requested a full tablet report, sending...
I20260621 06:51:42.871975 21015 ts_manager.cc:194] Registered new tserver with Master: 90f7693c1df54bac9bd2b7326db1a3a5 (127.20.120.193:43167)
I20260621 06:51:42.873632 21015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:57525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:51:42.893225 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45661
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:43.201141 21209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:43.201395 21209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:43.201570 21209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:43.208303 21209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:43.208416 21209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:43.208598 21209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:51:43.217839 21209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45661
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:43.218930 21209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:43.220139 21209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:43.231935 21215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:43.232019 21214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:43.232147 21217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:43.233322 21209 server_base.cc:1034] running on GCE node
I20260621 06:51:43.233804 21209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:43.234721 21209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:43.235941 21209 hybrid_clock.cc:648] HybridClock initialized: now 1782024703235899 us; error 46 us; skew 500 ppm
I20260621 06:51:43.236244 21209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:43 dist-test-slave-gmf2 krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024703, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:43.239082 21209 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:51:43.240497 21209 webserver.cc:460] Webserver started at http://127.20.120.194:38147/ using document root <none> and password file <none>
I20260621 06:51:43.241175 21209 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:43.241317 21209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:43.241619 21209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:43.244421 21209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "ac105fa887174492a296068b5d7e4c11"
format_stamp: "Formatted at 2026-06-21 06:51:43 on dist-test-slave-gmf2"
server_key: "7551a0c5fc07e5af2832fcd9d6fd86d0"
server_key_iv: "9a851d2b65eea9f41c2e0dedf55bc21b"
server_key_version: "encryptionkey@0"
I20260621 06:51:43.245153 21209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "ac105fa887174492a296068b5d7e4c11"
format_stamp: "Formatted at 2026-06-21 06:51:43 on dist-test-slave-gmf2"
server_key: "7551a0c5fc07e5af2832fcd9d6fd86d0"
server_key_iv: "9a851d2b65eea9f41c2e0dedf55bc21b"
server_key_version: "encryptionkey@0"
I20260621 06:51:43.249725 21209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260621 06:51:43.253481 21224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:43.254752 21209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:51:43.254920 21209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "ac105fa887174492a296068b5d7e4c11"
format_stamp: "Formatted at 2026-06-21 06:51:43 on dist-test-slave-gmf2"
server_key: "7551a0c5fc07e5af2832fcd9d6fd86d0"
server_key_iv: "9a851d2b65eea9f41c2e0dedf55bc21b"
server_key_version: "encryptionkey@0"
I20260621 06:51:43.255108 21209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:43.276844 21209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:43.281225 21209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:43.281522 21209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:43.282457 21209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:51:43.284130 21209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:51:43.284243 21209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260621 06:51:43.284400 21209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:51:43.284490 21209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:43.323470 21209 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:42053
I20260621 06:51:43.323503 21336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:42053 every 8 connection(s)
I20260621 06:51:43.324692 21209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:51:43.326272 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 21209
I20260621 06:51:43.326475 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:51:43.327018 20963 external_mini_cluster.cc:1428] Setting key 5f7b8aefd62dcf850218d6f3fcd7acfa
Jun 21 06:51:43 dist-test-slave-gmf2 krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024703, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
I20260621 06:51:43.344388 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:43.327158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:45319 (local address 127.20.120.254:45661)
0621 06:51:43.327319 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:43.327323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:43.328625 (+ 1302us) server_negotiation.cc:407] Connection header received
0621 06:51:43.330538 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:43.330546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:43.330630 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:43.330742 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:43.332425 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.333340 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.335096 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.335873 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.336008 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:43.339212 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:43.339216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:43.339238 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:43.339275 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:43.341674 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:43.342452 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:43.342456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:43.342457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:43.342589 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:43.342988 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:43.342991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:43.342993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:43.343216 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:43.343335 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:43.343955 (+ 620us) server_negotiation.cc:299] Negotiation successful
0621 06:51:43.344140 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 06:51:43.345254 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:43.327723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45661 (local address 127.20.120.194:45319)
0621 06:51:43.328453 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:43.328502 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:43.330299 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:43.330918 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:43.330925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:43.331765 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:43.332266 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:43.332282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.333492 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.333498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:43.334953 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:43.334974 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.336072 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.336077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:43.336250 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:43.336888 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:43.336907 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:43.339032 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:43.341805 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:43.341823 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:43.341835 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:43.342353 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:43.342690 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:43.342697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:43.342702 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:43.342891 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:43.343333 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:43.343339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:43.343655 (+ 316us) client_negotiation.cc:769] Sending connection context
0621 06:51:43.343857 (+ 202us) client_negotiation.cc:240] Negotiation successful
0621 06:51:43.344198 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":281,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
I20260621 06:51:43.346642 21337 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45661
I20260621 06:51:43.346926 21337 heartbeater.cc:464] Registering TS with master...
I20260621 06:51:43.347575 21337 heartbeater.cc:511] Master 127.20.120.254:45661 requested a full tablet report, sending...
I20260621 06:51:43.349247 21015 ts_manager.cc:194] Registered new tserver with Master: ac105fa887174492a296068b5d7e4c11 (127.20.120.194:42053)
I20260621 06:51:43.350103 21015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:45319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:51:43.365140 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45661
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:43.675660 21342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:43.675913 21342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:43.676002 21342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:43.683454 21342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:43.683547 21342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:43.683650 21342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:51:43.692433 21342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45661
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:43.693547 21342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:43.694788 21342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:43.708034 21347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:43.707865 21348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:43.708195 21350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:43.709002 21342 server_base.cc:1034] running on GCE node
I20260621 06:51:43.709530 21342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:43.710418 21342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:43.711634 21342 hybrid_clock.cc:648] HybridClock initialized: now 1782024703711608 us; error 41 us; skew 500 ppm
I20260621 06:51:43.711915 21342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:43 dist-test-slave-gmf2 krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024703, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:43.714730 21342 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:51:43.716311 21342 webserver.cc:460] Webserver started at http://127.20.120.195:40355/ using document root <none> and password file <none>
I20260621 06:51:43.716913 21342 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:43.717092 21342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:43.717394 21342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:43.720057 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "e893f434ebed4e53a446be3f3735d0fd"
format_stamp: "Formatted at 2026-06-21 06:51:43 on dist-test-slave-gmf2"
server_key: "49703c3993614bc15a5a34a478ea0621"
server_key_iv: "85065cbf9b4e4a975217dfda6c134412"
server_key_version: "encryptionkey@0"
I20260621 06:51:43.720810 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "e893f434ebed4e53a446be3f3735d0fd"
format_stamp: "Formatted at 2026-06-21 06:51:43 on dist-test-slave-gmf2"
server_key: "49703c3993614bc15a5a34a478ea0621"
server_key_iv: "85065cbf9b4e4a975217dfda6c134412"
server_key_version: "encryptionkey@0"
I20260621 06:51:43.725379 21342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260621 06:51:43.728466 21357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:43.729636 21342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 06:51:43.729794 21342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "e893f434ebed4e53a446be3f3735d0fd"
format_stamp: "Formatted at 2026-06-21 06:51:43 on dist-test-slave-gmf2"
server_key: "49703c3993614bc15a5a34a478ea0621"
server_key_iv: "85065cbf9b4e4a975217dfda6c134412"
server_key_version: "encryptionkey@0"
I20260621 06:51:43.729998 21342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:43.751597 21342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:43.755820 21342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:43.756081 21342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:43.756951 21342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:51:43.758306 21342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:51:43.758402 21342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:43.758541 21342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:51:43.758610 21342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:43.790885 21342 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:42457
I20260621 06:51:43.790959 21469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:42457 every 8 connection(s)
I20260621 06:51:43.792093 21342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:51:43.799528 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 21342
I20260621 06:51:43.799700 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:51:43.800074 20963 external_mini_cluster.cc:1428] Setting key 635a1613b94b61eb70701e8e52c02c0b
Jun 21 06:51:43 dist-test-slave-gmf2 krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024703, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:43.810561 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:43.794376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:52429 (local address 127.20.120.254:45661)
0621 06:51:43.794549 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:43.794554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:43.795630 (+ 1076us) server_negotiation.cc:407] Connection header received
0621 06:51:43.797333 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:43.797340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:43.797425 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:43.797527 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:43.799393 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.800139 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.802039 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.802576 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.802670 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:43.806039 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:43.806043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:43.806047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:43.806076 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:43.808257 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:43.808865 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:43.808868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:43.808870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:43.808986 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:43.809317 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:43.809320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:43.809322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:43.809544 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:43.809648 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:43.810218 (+ 570us) server_negotiation.cc:299] Negotiation successful
0621 06:51:43.810358 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 06:51:43.811364 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:43.794836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45661 (local address 127.20.120.195:52429)
0621 06:51:43.795489 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:43.795527 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:43.797086 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:43.797772 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:43.797784 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:43.798263 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:43.799213 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:43.799238 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.800275 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.800280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:43.801837 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:43.801848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.802777 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.802785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:43.803074 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:43.803801 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:43.803822 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:43.805788 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:43.808370 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:43.808376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:43.808386 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:43.808771 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:43.809080 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:43.809083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:43.809085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:43.809235 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:43.809635 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:43.809639 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:43.809895 (+ 256us) client_negotiation.cc:769] Sending connection context
0621 06:51:43.810152 (+ 257us) client_negotiation.cc:240] Negotiation successful
0621 06:51:43.810459 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":244,"threads_started":1}
I20260621 06:51:43.812510 21470 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45661
I20260621 06:51:43.812741 21470 heartbeater.cc:464] Registering TS with master...
I20260621 06:51:43.813304 21470 heartbeater.cc:511] Master 127.20.120.254:45661 requested a full tablet report, sending...
I20260621 06:51:43.814729 21014 ts_manager.cc:194] Registered new tserver with Master: e893f434ebed4e53a446be3f3735d0fd (127.20.120.195:42457)
I20260621 06:51:43.815622 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:51:43.815702 21014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:52429
Jun 21 06:51:43 dist-test-slave-gmf2 krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024703, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 06:51:43 dist-test-slave-gmf2 krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024703, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:51:43.860615 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:43.844144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46882 (local address 127.20.120.193:43167)
0621 06:51:43.845115 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:43.845121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:43.845147 (+ 26us) server_negotiation.cc:407] Connection header received
0621 06:51:43.845198 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:43.845204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:43.845392 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:43.845516 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:43.847334 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.848356 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.849679 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.850237 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.850329 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:43.854768 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:43.854783 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:43.854798 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:43.854839 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:43.858257 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:43.858800 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:43.858807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:43.858809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:43.858967 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:43.859441 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:43.859444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:43.859446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:43.859739 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:43.859931 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:43.860189 (+ 258us) server_negotiation.cc:299] Negotiation successful
0621 06:51:43.860350 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":685,"threads_started":1}
W20260621 06:51:43.861886 21119 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46882
I20260621 06:51:43.876382 21204 heartbeater.cc:503] Master 127.20.120.254:45661 was elected leader, sending a full tablet report...
Jun 21 06:51:43 dist-test-slave-gmf2 krb5kdc[20970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 06:51:43 dist-test-slave-gmf2 krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:51:43.902787 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:43.889596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.20.120.193:43167)
0621 06:51:43.889783 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:43.889790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:43.890143 (+ 353us) server_negotiation.cc:407] Connection header received
0621 06:51:43.890314 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:43.890319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:43.890383 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:43.890478 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:43.891665 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.892348 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.893540 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.894014 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.894139 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:43.898045 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:43.898050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:43.898053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:43.898085 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:43.900529 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:43.901095 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:43.901103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:43.901109 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:43.901282 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:43.901607 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:43.901613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:43.901618 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:43.901986 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:43.902177 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:43.902403 (+ 226us) server_negotiation.cc:299] Negotiation successful
0621 06:51:43.902562 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 06:51:43.904279 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 21076
I20260621 06:51:43.916458 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:43.905324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.20.120.193:43167)
0621 06:51:43.905532 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:43.905539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:43.905751 (+ 212us) server_negotiation.cc:407] Connection header received
0621 06:51:43.905985 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:43.905989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:43.906038 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:43.906131 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:43.907562 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.908191 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.909335 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:43.909790 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:43.909941 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:43.911995 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:43.912000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:43.912004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:43.912029 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:43.914155 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:43.914740 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:43.914746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:43.914750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:43.914887 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:43.915212 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:43.915217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:43.915221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:43.915443 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:43.915652 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:43.915918 (+ 266us) server_negotiation.cc:299] Negotiation successful
0621 06:51:43.916108 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 06:51:43.917191 21199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:51:43.996411 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 21076
I20260621 06:51:44.013707 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 21209
Jun 21 06:51:44 dist-test-slave-gmf2 krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:51:44.028743 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:44.014603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.20.120.194:42053)
0621 06:51:44.015096 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:44.015103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:44.015128 (+ 25us) server_negotiation.cc:407] Connection header received
0621 06:51:44.015192 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:44.015198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:44.015396 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:44.015528 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:44.016590 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:44.017509 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:44.018704 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:44.019214 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:44.019311 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:44.023559 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:44.023570 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:44.023583 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:44.023619 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:44.026252 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:44.026863 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:44.026869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:44.026872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:44.026992 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:44.027331 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:44.027334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:44.027336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:44.027623 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:44.027846 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:44.028280 (+ 434us) server_negotiation.cc:299] Negotiation successful
0621 06:51:44.028457 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":263,"threads_started":1}
I20260621 06:51:44.030112 21332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:51:44.109465 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 21209
I20260621 06:51:44.129944 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 21342
Jun 21 06:51:44 dist-test-slave-gmf2 krb5kdc[20970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:51:44.145823 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:44.130913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.20.120.195:42457)
0621 06:51:44.131499 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:44.131505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:44.131551 (+ 46us) server_negotiation.cc:407] Connection header received
0621 06:51:44.131625 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:44.131630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:44.131835 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:44.131953 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:44.133197 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:44.134477 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:44.136035 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:44.136534 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:44.136620 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:44.140902 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:44.140912 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:44.140925 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:44.140956 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:44.143626 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:44.144242 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:44.144247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:44.144250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:44.144362 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:44.144670 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:44.144673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:44.144675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:44.144961 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:44.145182 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:44.145390 (+ 208us) server_negotiation.cc:299] Negotiation successful
0621 06:51:44.145563 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":301,"threads_started":1}
I20260621 06:51:44.147049 21465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:51:44.225493 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 21342
I20260621 06:51:44.241629 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 20983
I20260621 06:51:44.242780 21045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:51:44.338666 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 20983
2026-06-21T06:51:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3664 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:51:44 dist-test-slave-gmf2 krb5kdc[21491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:51:44 dist-test-slave-gmf2 krb5kdc[21491](info): set up 2 sockets
Jun 21 06:51:44 dist-test-slave-gmf2 krb5kdc[21491](info): commencing operation
krb5kdc: starting...
W20260621 06:51:45.398031 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:51:45 dist-test-slave-gmf2 krb5kdc[21491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:51:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:51:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:51:45.579713 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:36201
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:36201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:45.893400 21504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:45.893710 21504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:45.893834 21504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:45.901260 21504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:51:45.901393 21504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:45.901443 21504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:45.901492 21504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:51:45.901559 21504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:51:45.911792 21504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:36201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:36201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:45.913196 21504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:45.914595 21504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:45.926893 21509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:45.926842 21510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:45.926923 21512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:45.927742 21504 server_base.cc:1034] running on GCE node
I20260621 06:51:45.928579 21504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:45.930179 21504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:45.931404 21504 hybrid_clock.cc:648] HybridClock initialized: now 1782024705931361 us; error 60 us; skew 500 ppm
I20260621 06:51:45.931773 21504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:45 dist-test-slave-gmf2 krb5kdc[21491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024705, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:45.934913 21504 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:51:45.936472 21504 webserver.cc:460] Webserver started at http://127.20.120.254:39101/ using document root <none> and password file <none>
I20260621 06:51:45.937171 21504 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:45.937316 21504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:45.937638 21504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:45.940625 21504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "01abc21b532c43f593a42fa0fc8737b3"
format_stamp: "Formatted at 2026-06-21 06:51:45 on dist-test-slave-gmf2"
server_key: "b7399ee08893d5c53d2fa40eb2837e52"
server_key_iv: "1d1bff0575dc33a5dc2b44f03e432fa5"
server_key_version: "encryptionkey@0"
I20260621 06:51:45.941371 21504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "01abc21b532c43f593a42fa0fc8737b3"
format_stamp: "Formatted at 2026-06-21 06:51:45 on dist-test-slave-gmf2"
server_key: "b7399ee08893d5c53d2fa40eb2837e52"
server_key_iv: "1d1bff0575dc33a5dc2b44f03e432fa5"
server_key_version: "encryptionkey@0"
I20260621 06:51:45.947188 21504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 06:51:45.950759 21519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:45.952069 21504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:51:45.952252 21504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "01abc21b532c43f593a42fa0fc8737b3"
format_stamp: "Formatted at 2026-06-21 06:51:45 on dist-test-slave-gmf2"
server_key: "b7399ee08893d5c53d2fa40eb2837e52"
server_key_iv: "1d1bff0575dc33a5dc2b44f03e432fa5"
server_key_version: "encryptionkey@0"
I20260621 06:51:45.952452 21504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:45.969390 21504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:45.973441 21504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:45.973743 21504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:45.992347 21504 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:36201
I20260621 06:51:45.992378 21570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:36201 every 8 connection(s)
I20260621 06:51:45.993734 21504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:51:45.995975 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 21504
I20260621 06:51:45.996206 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:51:45.996837 20963 external_mini_cluster.cc:1428] Setting key 9d13b4caa2b9ffef17058e2498a95478
I20260621 06:51:45.997668 21571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:46.011900 21571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3: Bootstrap starting.
Jun 21 06:51:46 dist-test-slave-gmf2 krb5kdc[21491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:46.015769 21571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:46.016865 21571 log.cc:824] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:46.020351 21571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3: No bootstrap required, opened a new log
I20260621 06:51:46.021169 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:45.998942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.20.120.254:36201)
0621 06:51:45.999706 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:45.999720 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:45.999770 (+ 50us) server_negotiation.cc:407] Connection header received
0621 06:51:46.002417 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:46.002446 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:46.002968 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:46.003427 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:46.004727 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:46.006710 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:46.007915 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:46.009026 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:46.009225 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:46.013557 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:46.013565 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:46.013580 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:46.013622 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:46.016729 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:46.017626 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:46.017640 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:46.017650 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:46.017868 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:46.018410 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:46.018414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:46.018416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:46.018923 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:46.019139 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:46.019702 (+ 563us) server_negotiation.cc:299] Negotiation successful
0621 06:51:46.020065 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":324,"threads_started":1}
I20260621 06:51:46.028410 21571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abc21b532c43f593a42fa0fc8737b3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36201 } }
I20260621 06:51:46.028904 21571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:46.029017 21571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01abc21b532c43f593a42fa0fc8737b3, State: Initialized, Role: FOLLOWER
I20260621 06:51:46.029656 21571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abc21b532c43f593a42fa0fc8737b3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36201 } }
I20260621 06:51:46.029894 21571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:51:46.030090 21571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:51:46.030272 21571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:46.032467 21571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abc21b532c43f593a42fa0fc8737b3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36201 } }
I20260621 06:51:46.032984 21571 leader_election.cc:302] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01abc21b532c43f593a42fa0fc8737b3; no voters:
I20260621 06:51:46.033546 21571 leader_election.cc:288] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:51:46.033672 21576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:51:46.034464 21576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [term 1 LEADER]: Becoming Leader. State: Replica: 01abc21b532c43f593a42fa0fc8737b3, State: Running, Role: LEADER
I20260621 06:51:46.035075 21576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abc21b532c43f593a42fa0fc8737b3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36201 } }
I20260621 06:51:46.035949 21571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:51:46.038184 21577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01abc21b532c43f593a42fa0fc8737b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abc21b532c43f593a42fa0fc8737b3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36201 } } }
I20260621 06:51:46.038375 21578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01abc21b532c43f593a42fa0fc8737b3. Latest consensus state: current_term: 1 leader_uuid: "01abc21b532c43f593a42fa0fc8737b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abc21b532c43f593a42fa0fc8737b3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36201 } } }
I20260621 06:51:46.038615 21577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [sys.catalog]: This master's current role is: LEADER
I20260621 06:51:46.038654 21578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3 [sys.catalog]: This master's current role is: LEADER
I20260621 06:51:46.040150 21585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:51:46.045367 21585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:51:46.054149 21585 catalog_manager.cc:1347] Generated new cluster ID: dc577d51ed654f25a83e961b31199aa5
I20260621 06:51:46.054291 21585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:51:46.064733 21585 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:51:46.066260 21585 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:51:46.079734 21585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01abc21b532c43f593a42fa0fc8737b3: Generated new TSK 0
I20260621 06:51:46.080581 21585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:51:46.130648 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:36201
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:46.461864 21597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:46.462152 21597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:46.462330 21597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:46.469440 21597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:46.469586 21597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:46.469782 21597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:51:46.479744 21597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:36201
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:46.481024 21597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:46.482327 21597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:46.494611 21602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:46.494647 21605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:46.494699 21603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:46.495585 21597 server_base.cc:1034] running on GCE node
I20260621 06:51:46.496133 21597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:46.497037 21597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:46.498253 21597 hybrid_clock.cc:648] HybridClock initialized: now 1782024706498215 us; error 52 us; skew 500 ppm
I20260621 06:51:46.498558 21597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:46 dist-test-slave-gmf2 krb5kdc[21491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024706, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:46.501446 21597 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:51:46.502771 21597 webserver.cc:460] Webserver started at http://127.20.120.193:38903/ using document root <none> and password file <none>
I20260621 06:51:46.503392 21597 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:46.503536 21597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:46.503850 21597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:46.506672 21597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "a8754fab42924ab292ee9f3692a12115"
format_stamp: "Formatted at 2026-06-21 06:51:46 on dist-test-slave-gmf2"
server_key: "91325c2972afa1b0c9419bf5f195bb04"
server_key_iv: "04178046ac226a402330275bd7cd1b5e"
server_key_version: "encryptionkey@0"
I20260621 06:51:46.507431 21597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "a8754fab42924ab292ee9f3692a12115"
format_stamp: "Formatted at 2026-06-21 06:51:46 on dist-test-slave-gmf2"
server_key: "91325c2972afa1b0c9419bf5f195bb04"
server_key_iv: "04178046ac226a402330275bd7cd1b5e"
server_key_version: "encryptionkey@0"
I20260621 06:51:46.512231 21597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 06:51:46.515484 21612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:46.516762 21597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 06:51:46.516929 21597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "a8754fab42924ab292ee9f3692a12115"
format_stamp: "Formatted at 2026-06-21 06:51:46 on dist-test-slave-gmf2"
server_key: "91325c2972afa1b0c9419bf5f195bb04"
server_key_iv: "04178046ac226a402330275bd7cd1b5e"
server_key_version: "encryptionkey@0"
I20260621 06:51:46.517141 21597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:46.560606 21597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:46.565385 21597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:46.565749 21597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:46.567008 21597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:51:46.568960 21597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:51:46.569063 21597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:46.569238 21597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:51:46.569346 21597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:46.601460 21597 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:40909
I20260621 06:51:46.601521 21724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:40909 every 8 connection(s)
I20260621 06:51:46.602844 21597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:51:46.608412 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 21597
I20260621 06:51:46.608631 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:51:46.609437 20963 external_mini_cluster.cc:1428] Setting key bb18760358858b9ae36bb1dfdbbf912e
Jun 21 06:51:46 dist-test-slave-gmf2 krb5kdc[21491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024706, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:46.625281 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:46.605381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:37379 (local address 127.20.120.254:36201)
0621 06:51:46.605822 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:46.605826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:46.606740 (+ 914us) server_negotiation.cc:407] Connection header received
0621 06:51:46.608558 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:46.608564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:46.608653 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:46.608809 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:46.610723 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:46.611569 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:46.614171 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:46.614610 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:46.614732 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:46.618548 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:46.618558 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:46.618566 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:46.618607 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:46.622154 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:46.622834 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:46.622842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:46.622849 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:46.622996 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:46.623416 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:46.623423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:46.623426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:46.623815 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:46.623982 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:46.624613 (+ 631us) server_negotiation.cc:299] Negotiation successful
0621 06:51:46.624862 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":243,"threads_started":1}
I20260621 06:51:46.626010 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:46.605869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:36201 (local address 127.20.120.193:37379)
0621 06:51:46.606590 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:46.606626 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:46.608293 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:46.609077 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:46.609090 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:46.609830 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:46.610546 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:46.610571 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:46.611722 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:46.611727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:46.614000 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:46.614018 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:46.614845 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:46.614853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:46.615175 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:46.615899 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:46.615933 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:46.618325 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:46.622293 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:46.622299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:46.622310 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:46.622700 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:46.623116 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:46.623119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:46.623121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:46.623292 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:46.623937 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:46.623943 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:46.624232 (+ 289us) client_negotiation.cc:769] Sending connection context
0621 06:51:46.624443 (+ 211us) client_negotiation.cc:240] Negotiation successful
0621 06:51:46.624815 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":255,"threads_started":1}
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
I20260621 06:51:46.627590 21725 heartbeater.cc:347] Connected to a master server at 127.20.120.254:36201
I20260621 06:51:46.627861 21725 heartbeater.cc:464] Registering TS with master...
I20260621 06:51:46.628544 21725 heartbeater.cc:511] Master 127.20.120.254:36201 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
I20260621 06:51:46.630744 21536 ts_manager.cc:194] Registered new tserver with Master: a8754fab42924ab292ee9f3692a12115 (127.20.120.193:40909)
I20260621 06:51:46.632925 21536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:37379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:51:46.652357 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:36201
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:46.966851 21731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:46.967121 21731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:46.967216 21731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:46.974346 21731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:46.974450 21731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:46.974568 21731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:51:46.984050 21731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:36201
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:46.985249 21731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:46.986680 21731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:46.999338 21739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:46.999423 21736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:46.999396 21737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:47.000349 21731 server_base.cc:1034] running on GCE node
I20260621 06:51:47.000936 21731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:47.001888 21731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:47.003168 21731 hybrid_clock.cc:648] HybridClock initialized: now 1782024707003109 us; error 45 us; skew 500 ppm
I20260621 06:51:47.003525 21731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:47 dist-test-slave-gmf2 krb5kdc[21491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024707, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:47.007213 21731 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:51:47.008764 21731 webserver.cc:460] Webserver started at http://127.20.120.194:37169/ using document root <none> and password file <none>
I20260621 06:51:47.009423 21731 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:47.009572 21731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:47.009876 21731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:47.012681 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "4e109bc300734ea6ba29731b993a84a0"
format_stamp: "Formatted at 2026-06-21 06:51:47 on dist-test-slave-gmf2"
server_key: "9a006282b4cbebd99ed2f691c8dd2342"
server_key_iv: "f10654f07dd1ce9fa4134a333d09d2d4"
server_key_version: "encryptionkey@0"
I20260621 06:51:47.013433 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "4e109bc300734ea6ba29731b993a84a0"
format_stamp: "Formatted at 2026-06-21 06:51:47 on dist-test-slave-gmf2"
server_key: "9a006282b4cbebd99ed2f691c8dd2342"
server_key_iv: "f10654f07dd1ce9fa4134a333d09d2d4"
server_key_version: "encryptionkey@0"
I20260621 06:51:47.018626 21731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260621 06:51:47.022300 21746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:47.023633 21731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:51:47.023805 21731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "4e109bc300734ea6ba29731b993a84a0"
format_stamp: "Formatted at 2026-06-21 06:51:47 on dist-test-slave-gmf2"
server_key: "9a006282b4cbebd99ed2f691c8dd2342"
server_key_iv: "f10654f07dd1ce9fa4134a333d09d2d4"
server_key_version: "encryptionkey@0"
I20260621 06:51:47.024010 21731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:47.037988 21731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:47.042440 21731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:47.042745 21731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:47.043668 21731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:51:47.044936 21731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:51:47.045042 21731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:47.045188 21731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:51:47.045264 21731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:47.077790 21731 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:34807
I20260621 06:51:47.077811 21858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:34807 every 8 connection(s)
I20260621 06:51:47.079084 21731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:51:47.088222 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 21731
I20260621 06:51:47.088411 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:51:47.088928 20963 external_mini_cluster.cc:1428] Setting key b02a48a89ee1c1f3b4f8dcbbe2f70968
Jun 21 06:51:47 dist-test-slave-gmf2 krb5kdc[21491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024707, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:47.099442 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.081720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:47839 (local address 127.20.120.254:36201)
0621 06:51:47.081956 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:47.081962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:47.083086 (+ 1124us) server_negotiation.cc:407] Connection header received
0621 06:51:47.084775 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:47.084779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:47.084844 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:47.084931 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:47.086380 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.087060 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.088517 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.089136 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.089258 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.093149 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:47.093158 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:47.093165 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:47.093208 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:47.096350 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:47.097243 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:47.097247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:47.097249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:47.097364 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:47.097817 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:47.097820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:47.097822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:47.098126 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:47.098232 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:47.099063 (+ 831us) server_negotiation.cc:299] Negotiation successful
0621 06:51:47.099219 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 06:51:47.100997 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.082270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:36201 (local address 127.20.120.194:47839)
0621 06:51:47.082931 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:47.082968 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:47.084540 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:47.085121 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:47.085129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:47.085624 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:47.086242 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.086259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.087191 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.087195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.088367 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.088379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.089339 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.089344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.089602 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.090381 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:47.090419 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:47.092935 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:47.096497 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:47.096506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:47.096524 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:47.097130 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:47.097463 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:47.097471 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:47.097478 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:47.097712 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:47.098244 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:47.098255 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:47.098696 (+ 441us) client_negotiation.cc:769] Sending connection context
0621 06:51:47.098988 (+ 292us) client_negotiation.cc:240] Negotiation successful
0621 06:51:47.099553 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":252,"threads_started":1}
I20260621 06:51:47.102514 21859 heartbeater.cc:347] Connected to a master server at 127.20.120.254:36201
I20260621 06:51:47.102816 21859 heartbeater.cc:464] Registering TS with master...
I20260621 06:51:47.103574 21859 heartbeater.cc:511] Master 127.20.120.254:36201 requested a full tablet report, sending...
I20260621 06:51:47.104969 21536 ts_manager.cc:194] Registered new tserver with Master: 4e109bc300734ea6ba29731b993a84a0 (127.20.120.194:34807)
I20260621 06:51:47.105762 21536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:47839
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:51:47.129025 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:36201
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:47.462136 21864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:47.462472 21864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:47.462579 21864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:47.470073 21864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:47.470187 21864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:47.470302 21864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:51:47.480274 21864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:36201
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:47.481556 21864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:47.482863 21864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:47.496210 21869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:47.496315 21872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:47.496244 21870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:47.497263 21864 server_base.cc:1034] running on GCE node
I20260621 06:51:47.497861 21864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:47.498816 21864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:47.500054 21864 hybrid_clock.cc:648] HybridClock initialized: now 1782024707500014 us; error 42 us; skew 500 ppm
I20260621 06:51:47.500458 21864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:47 dist-test-slave-gmf2 krb5kdc[21491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024707, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:47.503582 21864 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:51:47.505033 21864 webserver.cc:460] Webserver started at http://127.20.120.195:36351/ using document root <none> and password file <none>
I20260621 06:51:47.505733 21864 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:47.505874 21864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:47.506219 21864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:47.509135 21864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "8f925fb512ee4503bf480bfdc0e7935d"
format_stamp: "Formatted at 2026-06-21 06:51:47 on dist-test-slave-gmf2"
server_key: "20d81d27ec9fc5a8c1d4466f6f519e32"
server_key_iv: "5058a6b55f0c9552cafb55e465e8e04c"
server_key_version: "encryptionkey@0"
I20260621 06:51:47.509987 21864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "8f925fb512ee4503bf480bfdc0e7935d"
format_stamp: "Formatted at 2026-06-21 06:51:47 on dist-test-slave-gmf2"
server_key: "20d81d27ec9fc5a8c1d4466f6f519e32"
server_key_iv: "5058a6b55f0c9552cafb55e465e8e04c"
server_key_version: "encryptionkey@0"
I20260621 06:51:47.515060 21864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 06:51:47.518362 21879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:47.519630 21864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:51:47.519798 21864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "8f925fb512ee4503bf480bfdc0e7935d"
format_stamp: "Formatted at 2026-06-21 06:51:47 on dist-test-slave-gmf2"
server_key: "20d81d27ec9fc5a8c1d4466f6f519e32"
server_key_iv: "5058a6b55f0c9552cafb55e465e8e04c"
server_key_version: "encryptionkey@0"
I20260621 06:51:47.519994 21864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:47.538964 21864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:47.543967 21864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:47.544306 21864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:47.545542 21864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:51:47.546833 21864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:51:47.546901 21864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:47.546985 21864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:51:47.547036 21864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:47.580610 21864 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:39659
I20260621 06:51:47.580644 21991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:39659 every 8 connection(s)
I20260621 06:51:47.581992 21864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:51:47.586241 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 21864
I20260621 06:51:47.586432 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:51:47.586853 20963 external_mini_cluster.cc:1428] Setting key 0af2370dc6b5ef82ebfe6c45457bb418
Jun 21 06:51:47 dist-test-slave-gmf2 krb5kdc[21491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024707, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:47.601943 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.584462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:50045 (local address 127.20.120.254:36201)
0621 06:51:47.584639 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:47.584644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:47.585859 (+ 1215us) server_negotiation.cc:407] Connection header received
0621 06:51:47.587933 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:47.587938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:47.588036 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:47.588139 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:47.589574 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.590446 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.591976 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.592381 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.592518 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.596225 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:47.596231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:47.596234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:47.596265 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:47.598662 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:47.599388 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:47.599396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:47.599402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:47.599547 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:47.599982 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:47.599988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:47.599991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:47.600266 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:47.600423 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:47.601323 (+ 900us) server_negotiation.cc:299] Negotiation successful
0621 06:51:47.601558 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 06:51:47.603428 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.584995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:36201 (local address 127.20.120.195:50045)
0621 06:51:47.585681 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:47.585721 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:47.587679 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:47.588366 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:47.588374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:47.588870 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:47.589405 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.589421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.590606 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.590610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.591833 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.591844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.592619 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.592626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.592854 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.593460 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:47.593484 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:47.596011 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:47.598811 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:47.598821 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:47.598838 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:47.599263 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:47.599671 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:47.599677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:47.599680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:47.599867 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:47.600398 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:47.600406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:47.600837 (+ 431us) client_negotiation.cc:769] Sending connection context
0621 06:51:47.601196 (+ 359us) client_negotiation.cc:240] Negotiation successful
0621 06:51:47.601727 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":266,"threads_started":1}
I20260621 06:51:47.604737 21992 heartbeater.cc:347] Connected to a master server at 127.20.120.254:36201
I20260621 06:51:47.605031 21992 heartbeater.cc:464] Registering TS with master...
I20260621 06:51:47.605844 21992 heartbeater.cc:511] Master 127.20.120.254:36201 requested a full tablet report, sending...
I20260621 06:51:47.607295 21536 ts_manager.cc:194] Registered new tserver with Master: 8f925fb512ee4503bf480bfdc0e7935d (127.20.120.195:39659)
I20260621 06:51:47.608059 21536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:50045
I20260621 06:51:47.614071 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:51:47.632613 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.620085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.20.120.254:36201)
0621 06:51:47.620296 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:47.620306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:47.620677 (+ 371us) server_negotiation.cc:407] Connection header received
0621 06:51:47.620861 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:47.620864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:47.620924 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:47.621038 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:47.622886 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.623687 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.625011 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.625397 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.625488 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.627636 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:47.627641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:47.627644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:47.627687 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:47.630419 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:47.630964 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:47.630968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:47.630970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:47.631100 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:47.631384 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:47.631387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:47.631389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:47.631682 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:47.631857 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:47.632120 (+ 263us) server_negotiation.cc:299] Negotiation successful
0621 06:51:47.632264 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 06:51:47.636608 21725 heartbeater.cc:503] Master 127.20.120.254:36201 was elected leader, sending a full tablet report...
I20260621 06:51:47.639158 21536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:51:47.646358 21536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:51:47.671414 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.659843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.20.120.193:40909)
0621 06:51:47.662535 (+ 2692us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:47.662548 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:47.662583 (+ 35us) server_negotiation.cc:407] Connection header received
0621 06:51:47.662692 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:47.662703 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:47.663002 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:47.664492 (+ 1490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:47.664704 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.666787 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.668942 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.670148 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.670662 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.670827 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:47.670894 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 06:51:47.671055 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2501,"thread_start_us":2343,"threads_started":1}
I20260621 06:51:47.671424 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.659224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:40909 (local address 127.0.0.1:58790)
0621 06:51:47.660333 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:47.660352 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:47.660463 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:47.663913 (+ 3450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:47.663924 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:47.663958 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:47.664305 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.664320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.666942 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.666951 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.668777 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.668788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.670375 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.670381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.670674 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.670730 (+ 56us) client_negotiation.cc:769] Sending connection context
0621 06:51:47.670841 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 06:51:47.670963 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":123,"thread_start_us":4128,"threads_started":1}
I20260621 06:51:47.672072 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.658671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:39659 (local address 127.0.0.1:55044)
0621 06:51:47.659557 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:47.659601 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:47.659775 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:47.664933 (+ 5158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:47.664940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:47.664980 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:47.665411 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.665421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.667077 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.667085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.669387 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.669399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.671194 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.671200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.671381 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.671426 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 06:51:47.671519 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 06:51:47.671674 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":81,"thread_start_us":272,"threads_started":1}
I20260621 06:51:47.672375 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.658973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.20.120.195:39659)
0621 06:51:47.664321 (+ 5348us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:47.664330 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:47.664363 (+ 33us) server_negotiation.cc:407] Connection header received
0621 06:51:47.664456 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:47.664463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:47.664705 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:47.664915 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:47.665596 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.666920 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.669538 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.670978 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.671166 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.671326 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:47.671789 (+ 463us) server_negotiation.cc:299] Negotiation successful
0621 06:51:47.671938 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5158,"thread_start_us":5216,"threads_started":1}
I20260621 06:51:47.675383 21660 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca80b2c70fe4b9287a47c46e72686c6 (DEFAULT_TABLE table=test-table [id=bf55491886fd48429fa545fc68c3dc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:47.676581 21660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca80b2c70fe4b9287a47c46e72686c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:47.676214 21927 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca80b2c70fe4b9287a47c46e72686c6 (DEFAULT_TABLE table=test-table [id=bf55491886fd48429fa545fc68c3dc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:47.677767 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.660333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.20.120.194:34807)
0621 06:51:47.665909 (+ 5576us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:47.665916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:47.665942 (+ 26us) server_negotiation.cc:407] Connection header received
0621 06:51:47.666044 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:47.666050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:47.666255 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:47.666470 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:47.667339 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.668862 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.674723 (+ 5861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.676559 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.676737 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.676921 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:47.677266 (+ 345us) server_negotiation.cc:299] Negotiation successful
0621 06:51:47.677453 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5439,"thread_start_us":4905,"threads_started":1}
I20260621 06:51:47.677784 22006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.659818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:34807 (local address 127.0.0.1:49024)
0621 06:51:47.663601 (+ 3783us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:47.663617 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:47.663732 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:47.666669 (+ 2937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:47.666673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:47.666694 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:47.667109 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.667119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.669026 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.669031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.674540 (+ 5509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.674555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.676826 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.676831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.677030 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.677080 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 06:51:47.677185 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 06:51:47.677329 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3647,"mutex_wait_us":39,"thread_start_us":687,"threads_started":1}
I20260621 06:51:47.677815 21927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca80b2c70fe4b9287a47c46e72686c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:47.681372 21794 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca80b2c70fe4b9287a47c46e72686c6 (DEFAULT_TABLE table=test-table [id=bf55491886fd48429fa545fc68c3dc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:47.682741 21794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca80b2c70fe4b9287a47c46e72686c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:47.688459 22010 tablet_bootstrap.cc:492] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115: Bootstrap starting.
I20260621 06:51:47.692540 22010 tablet_bootstrap.cc:654] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:47.693792 22011 tablet_bootstrap.cc:492] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d: Bootstrap starting.
I20260621 06:51:47.694411 22010 log.cc:824] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:47.696921 22013 tablet_bootstrap.cc:492] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0: Bootstrap starting.
I20260621 06:51:47.697149 22011 tablet_bootstrap.cc:654] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:47.697618 22010 tablet_bootstrap.cc:492] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115: No bootstrap required, opened a new log
I20260621 06:51:47.697978 22010 ts_tablet_manager.cc:1397] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260621 06:51:47.698486 22011 log.cc:824] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:47.700665 22013 tablet_bootstrap.cc:654] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:47.701555 22011 tablet_bootstrap.cc:492] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d: No bootstrap required, opened a new log
I20260621 06:51:47.701865 22011 ts_tablet_manager.cc:1397] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260621 06:51:47.701949 22013 log.cc:824] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:47.705091 22013 tablet_bootstrap.cc:492] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0: No bootstrap required, opened a new log
I20260621 06:51:47.705360 22013 ts_tablet_manager.cc:1397] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260621 06:51:47.708647 22010 raft_consensus.cc:348] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f925fb512ee4503bf480bfdc0e7935d" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 39659 } } peers { permanent_uuid: "4e109bc300734ea6ba29731b993a84a0" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34807 } } peers { permanent_uuid: "a8754fab42924ab292ee9f3692a12115" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40909 } }
I20260621 06:51:47.709103 22010 raft_consensus.cc:374] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:47.709229 22010 raft_consensus.cc:729] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8754fab42924ab292ee9f3692a12115, State: Initialized, Role: FOLLOWER
I20260621 06:51:47.709926 22010 consensus_queue.cc:260] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f925fb512ee4503bf480bfdc0e7935d" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 39659 } } peers { permanent_uuid: "4e109bc300734ea6ba29731b993a84a0" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34807 } } peers { permanent_uuid: "a8754fab42924ab292ee9f3692a12115" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40909 } }
I20260621 06:51:47.712102 22010 ts_tablet_manager.cc:1428] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115: Time spent starting tablet: real 0.014s user 0.004s sys 0.008s
I20260621 06:51:47.712971 22011 raft_consensus.cc:348] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f925fb512ee4503bf480bfdc0e7935d" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 39659 } } peers { permanent_uuid: "4e109bc300734ea6ba29731b993a84a0" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34807 } } peers { permanent_uuid: "a8754fab42924ab292ee9f3692a12115" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40909 } }
I20260621 06:51:47.713467 22011 raft_consensus.cc:374] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:47.713245 22013 raft_consensus.cc:348] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f925fb512ee4503bf480bfdc0e7935d" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 39659 } } peers { permanent_uuid: "4e109bc300734ea6ba29731b993a84a0" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34807 } } peers { permanent_uuid: "a8754fab42924ab292ee9f3692a12115" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40909 } }
I20260621 06:51:47.713613 22011 raft_consensus.cc:729] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f925fb512ee4503bf480bfdc0e7935d, State: Initialized, Role: FOLLOWER
I20260621 06:51:47.713721 22013 raft_consensus.cc:374] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:47.713862 22013 raft_consensus.cc:729] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e109bc300734ea6ba29731b993a84a0, State: Initialized, Role: FOLLOWER
I20260621 06:51:47.714322 22011 consensus_queue.cc:260] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f925fb512ee4503bf480bfdc0e7935d" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 39659 } } peers { permanent_uuid: "4e109bc300734ea6ba29731b993a84a0" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34807 } } peers { permanent_uuid: "a8754fab42924ab292ee9f3692a12115" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40909 } }
I20260621 06:51:47.714531 22013 consensus_queue.cc:260] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f925fb512ee4503bf480bfdc0e7935d" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 39659 } } peers { permanent_uuid: "4e109bc300734ea6ba29731b993a84a0" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34807 } } peers { permanent_uuid: "a8754fab42924ab292ee9f3692a12115" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40909 } }
I20260621 06:51:47.715667 21992 heartbeater.cc:503] Master 127.20.120.254:36201 was elected leader, sending a full tablet report...
I20260621 06:51:47.716068 22011 ts_tablet_manager.cc:1428] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260621 06:51:47.717371 21859 heartbeater.cc:503] Master 127.20.120.254:36201 was elected leader, sending a full tablet report...
I20260621 06:51:47.718040 22013 ts_tablet_manager.cc:1428] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
W20260621 06:51:47.831252 21860 tablet.cc:2369] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:51:47.833460 21993 tablet.cc:2369] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:51:47.855111 21726 tablet.cc:2369] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:51:47.963491 22017 raft_consensus.cc:482] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:51:47.963811 22017 raft_consensus.cc:504] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f925fb512ee4503bf480bfdc0e7935d" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 39659 } } peers { permanent_uuid: "4e109bc300734ea6ba29731b993a84a0" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34807 } } peers { permanent_uuid: "a8754fab42924ab292ee9f3692a12115" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40909 } }
I20260621 06:51:47.965448 22017 leader_election.cc:288] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e109bc300734ea6ba29731b993a84a0 (127.20.120.194:34807), a8754fab42924ab292ee9f3692a12115 (127.20.120.193:40909)
I20260621 06:51:47.971882 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.965916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:51319 (local address 127.20.120.193:40909)
0621 06:51:47.966160 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:47.966168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:47.966191 (+ 23us) server_negotiation.cc:407] Connection header received
0621 06:51:47.966341 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:47.966346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:47.966422 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:47.966535 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:47.967002 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.967970 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.970107 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.971026 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.971136 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.971184 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:47.971522 (+ 338us) server_negotiation.cc:299] Negotiation successful
0621 06:51:47.971614 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 06:51:47.971916 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.965715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:40909 (local address 127.20.120.195:51319)
0621 06:51:47.966015 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:47.966035 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:47.966177 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:47.966559 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:47.966562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:47.966585 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:47.966834 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.966845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.968107 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.968113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.969894 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.969970 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.971168 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.971175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.971354 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.971391 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 06:51:47.971491 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 06:51:47.971646 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"spinlock_wait_cycles":36736}
I20260621 06:51:47.972424 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.965805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:42653 (local address 127.20.120.194:34807)
0621 06:51:47.966037 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:47.966044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:47.966656 (+ 612us) server_negotiation.cc:407] Connection header received
0621 06:51:47.966838 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:47.966843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:47.966925 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:47.967030 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:47.967584 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.968795 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.970649 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.971577 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.971702 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.971767 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:47.972135 (+ 368us) server_negotiation.cc:299] Negotiation successful
0621 06:51:47.972243 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 06:51:47.972965 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:47.965717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:34807 (local address 127.20.120.195:42653)
0621 06:51:47.966551 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:47.966573 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:47.966714 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:47.967060 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:47.967065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:47.967090 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:47.967435 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.967446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.968934 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.968938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.970507 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:47.970521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:47.971760 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:47.971766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:47.971941 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:47.971977 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 06:51:47.972594 (+ 617us) client_negotiation.cc:240] Negotiation successful
0621 06:51:47.972712 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":31,"spinlock_wait_cycles":6784,"thread_start_us":455,"threads_started":1}
I20260621 06:51:47.972908 21680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca80b2c70fe4b9287a47c46e72686c6" candidate_uuid: "8f925fb512ee4503bf480bfdc0e7935d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8754fab42924ab292ee9f3692a12115" is_pre_election: true
I20260621 06:51:47.973417 21680 raft_consensus.cc:2454] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f925fb512ee4503bf480bfdc0e7935d in term 0.
I20260621 06:51:47.974149 21881 leader_election.cc:302] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f925fb512ee4503bf480bfdc0e7935d, a8754fab42924ab292ee9f3692a12115; no voters:
I20260621 06:51:47.974002 21814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca80b2c70fe4b9287a47c46e72686c6" candidate_uuid: "8f925fb512ee4503bf480bfdc0e7935d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e109bc300734ea6ba29731b993a84a0" is_pre_election: true
I20260621 06:51:47.974519 22017 raft_consensus.cc:2781] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:51:47.974529 21814 raft_consensus.cc:2454] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f925fb512ee4503bf480bfdc0e7935d in term 0.
I20260621 06:51:47.974664 22017 raft_consensus.cc:482] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:51:47.974779 22017 raft_consensus.cc:3037] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:47.976817 22017 raft_consensus.cc:504] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f925fb512ee4503bf480bfdc0e7935d" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 39659 } } peers { permanent_uuid: "4e109bc300734ea6ba29731b993a84a0" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34807 } } peers { permanent_uuid: "a8754fab42924ab292ee9f3692a12115" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40909 } }
I20260621 06:51:47.977412 22017 leader_election.cc:288] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [CANDIDATE]: Term 1 election: Requested vote from peers 4e109bc300734ea6ba29731b993a84a0 (127.20.120.194:34807), a8754fab42924ab292ee9f3692a12115 (127.20.120.193:40909)
I20260621 06:51:47.977682 21814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca80b2c70fe4b9287a47c46e72686c6" candidate_uuid: "8f925fb512ee4503bf480bfdc0e7935d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e109bc300734ea6ba29731b993a84a0"
I20260621 06:51:47.977856 21814 raft_consensus.cc:3037] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:47.977813 21680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca80b2c70fe4b9287a47c46e72686c6" candidate_uuid: "8f925fb512ee4503bf480bfdc0e7935d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8754fab42924ab292ee9f3692a12115"
I20260621 06:51:47.978034 21680 raft_consensus.cc:3037] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:47.980196 21814 raft_consensus.cc:2454] T 2ca80b2c70fe4b9287a47c46e72686c6 P 4e109bc300734ea6ba29731b993a84a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f925fb512ee4503bf480bfdc0e7935d in term 1.
I20260621 06:51:47.980518 21882 leader_election.cc:302] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e109bc300734ea6ba29731b993a84a0, 8f925fb512ee4503bf480bfdc0e7935d; no voters:
I20260621 06:51:47.980782 22017 raft_consensus.cc:2781] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:51:47.980892 21680 raft_consensus.cc:2454] T 2ca80b2c70fe4b9287a47c46e72686c6 P a8754fab42924ab292ee9f3692a12115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f925fb512ee4503bf480bfdc0e7935d in term 1.
I20260621 06:51:47.981269 22017 raft_consensus.cc:686] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [term 1 LEADER]: Becoming Leader. State: Replica: 8f925fb512ee4503bf480bfdc0e7935d, State: Running, Role: LEADER
I20260621 06:51:47.981731 22017 consensus_queue.cc:237] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f925fb512ee4503bf480bfdc0e7935d" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 39659 } } peers { permanent_uuid: "4e109bc300734ea6ba29731b993a84a0" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34807 } } peers { permanent_uuid: "a8754fab42924ab292ee9f3692a12115" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40909 } }
I20260621 06:51:47.986472 21536 catalog_manager.cc:5495] T 2ca80b2c70fe4b9287a47c46e72686c6 P 8f925fb512ee4503bf480bfdc0e7935d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f925fb512ee4503bf480bfdc0e7935d (127.20.120.195). New cstate: current_term: 1 leader_uuid: "8f925fb512ee4503bf480bfdc0e7935d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f925fb512ee4503bf480bfdc0e7935d" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 39659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e109bc300734ea6ba29731b993a84a0" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8754fab42924ab292ee9f3692a12115" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40909 } health_report { overall_health: UNKNOWN } } }
Jun 21 06:51:48 dist-test-slave-gmf2 krb5kdc[21491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:51:48.047333 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:48.029441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.20.120.193:40909)
0621 06:51:48.029643 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:48.029651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:48.030207 (+ 556us) server_negotiation.cc:407] Connection header received
0621 06:51:48.030337 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:48.030340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:48.030396 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:48.030473 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:48.032258 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:48.032971 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:48.034121 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:48.034515 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:48.034603 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:48.042423 (+ 7820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:48.042437 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:48.042453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:48.042514 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:48.045027 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:48.045670 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:48.045676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:48.045679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:48.045797 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:48.046142 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:48.046145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:48.046147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:48.046447 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:48.046659 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:48.046833 (+ 174us) server_negotiation.cc:299] Negotiation successful
0621 06:51:48.046970 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Jun 21 06:51:48 dist-test-slave-gmf2 krb5kdc[21491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 06:51:48 dist-test-slave-gmf2 krb5kdc[21491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:51:48.090219 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:48.076219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.20.120.193:40909)
0621 06:51:48.076428 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:48.076435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:48.076600 (+ 165us) server_negotiation.cc:407] Connection header received
0621 06:51:48.076737 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:48.076740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:48.076797 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:48.076897 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:48.078146 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:48.079306 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:48.080608 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:48.081234 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:48.081369 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:48.085392 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:48.085396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:48.085399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:48.085427 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:48.087982 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:48.088609 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:48.088612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:48.088615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:48.088715 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:48.089073 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:48.089075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:48.089077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:48.089285 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:48.089467 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:48.089715 (+ 248us) server_negotiation.cc:299] Negotiation successful
0621 06:51:48.089859 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Jun 21 06:51:48 dist-test-slave-gmf2 krb5kdc[21491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 06:51:48 dist-test-slave-gmf2 krb5kdc[21491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:51:48.133656 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:48.118770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58830 (local address 127.20.120.193:40909)
0621 06:51:48.118981 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:48.118989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:48.119252 (+ 263us) server_negotiation.cc:407] Connection header received
0621 06:51:48.119430 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:48.119441 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:48.119497 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:48.119618 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:48.121415 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:48.122220 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:48.123475 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:48.124039 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:48.124187 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:48.128141 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:48.128146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:48.128152 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:48.128206 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:48.130926 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:48.131580 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:48.131591 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:48.131611 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:48.131729 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:48.132116 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:48.132124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:48.132127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:48.132431 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:48.132677 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:48.133056 (+ 379us) server_negotiation.cc:299] Negotiation successful
0621 06:51:48.133293 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260621 06:51:48.135404 21640 tablet_service.cc:2908] Rejecting scan request for tablet 2ca80b2c70fe4b9287a47c46e72686c6: Uninitialized: clean time has not yet been initialized
I20260621 06:51:48.138023 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 21597
I20260621 06:51:48.149595 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:48.138927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.20.120.193:40909)
0621 06:51:48.139052 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:48.139056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:48.139436 (+ 380us) server_negotiation.cc:407] Connection header received
0621 06:51:48.139531 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:48.139569 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:48.139643 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:48.139715 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:48.141336 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:48.142011 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:48.143032 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:48.143401 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:48.143487 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:48.145451 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:48.145455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:48.145458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:48.145480 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:48.147711 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:48.148218 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:48.148222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:48.148224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:48.148317 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:48.148565 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:48.148567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:48.148569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:48.148794 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:48.148977 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:48.149157 (+ 180us) server_negotiation.cc:299] Negotiation successful
0621 06:51:48.149291 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260621 06:51:48.150178 21720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:51:48.236382 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 21597
I20260621 06:51:48.252444 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 21731
Jun 21 06:51:48 dist-test-slave-gmf2 krb5kdc[21491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:51:48.266255 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:48.253245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.20.120.194:34807)
0621 06:51:48.253415 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:48.253423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:48.253444 (+ 21us) server_negotiation.cc:407] Connection header received
0621 06:51:48.253648 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:48.253655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:48.253721 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:48.253847 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:48.255011 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:48.255732 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:48.256947 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:48.257316 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:48.257444 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:48.261059 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:48.261073 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:48.261088 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:48.261122 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:48.263631 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:48.264285 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:48.264293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:48.264295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:48.264419 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:48.264775 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:48.264781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:48.264785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:48.265140 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:48.265420 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:48.265699 (+ 279us) server_negotiation.cc:299] Negotiation successful
0621 06:51:48.265961 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 06:51:48.266865 21854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:51:48.351076 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 21731
I20260621 06:51:48.367992 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 21864
Jun 21 06:51:48 dist-test-slave-gmf2 krb5kdc[21491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:51:48.382091 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:48.368702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.20.120.195:39659)
0621 06:51:48.369213 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:48.369218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:48.369232 (+ 14us) server_negotiation.cc:407] Connection header received
0621 06:51:48.369282 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:48.369285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:48.369338 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:48.369421 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:48.370547 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:48.371224 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:48.372419 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:48.372785 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:48.372885 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:48.376696 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:48.376709 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:48.376725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:48.376768 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:48.379491 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:48.380133 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:48.380140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:48.380142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:48.380282 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:48.380716 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:48.380721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:48.380724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:48.381044 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:48.381242 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:48.381492 (+ 250us) server_negotiation.cc:299] Negotiation successful
0621 06:51:48.381663 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":294,"threads_started":1}
I20260621 06:51:48.382764 21987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:51:48.467870 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:48.467426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:40909 (local address 127.20.120.195:43765)
0621 06:51:48.467595 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:48.467738 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:40909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 06:51:48.467921 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 21864
I20260621 06:51:48.484148 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 21504
I20260621 06:51:48.485230 21566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:51:48.589574 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 21504
2026-06-21T06:51:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4256 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:51:48 dist-test-slave-gmf2 krb5kdc[22040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:51:48 dist-test-slave-gmf2 krb5kdc[22040](info): set up 2 sockets
Jun 21 06:51:48 dist-test-slave-gmf2 krb5kdc[22040](info): commencing operation
krb5kdc: starting...
W20260621 06:51:49.653944 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:51:49 dist-test-slave-gmf2 krb5kdc[22040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:51:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:51:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:51:49.824676 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:45775
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:39809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:45775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:50.132372 22053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:50.132637 22053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:50.132789 22053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:50.139729 22053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:51:50.139819 22053 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 06:51:50.139895 22053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:50.140002 22053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:50.140087 22053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:51:50.140178 22053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:51:50.149438 22053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:39809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:45775
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:45775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:50.150642 22053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:50.151927 22053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:50.163538 22058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:50.163472 22059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:50.163503 22061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:50.164547 22053 server_base.cc:1034] running on GCE node
I20260621 06:51:50.165217 22053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:50.166471 22053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:50.167699 22053 hybrid_clock.cc:648] HybridClock initialized: now 1782024710167629 us; error 50 us; skew 500 ppm
I20260621 06:51:50.168040 22053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:50 dist-test-slave-gmf2 krb5kdc[22040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024710, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:50.171002 22053 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:51:50.172391 22053 webserver.cc:460] Webserver started at http://127.20.120.254:40935/ using document root <none> and password file <none>
I20260621 06:51:50.173089 22053 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:50.173214 22053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:50.173524 22053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:50.176335 22053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "69f10dce80074ac691da28c98d82a6a4"
format_stamp: "Formatted at 2026-06-21 06:51:50 on dist-test-slave-gmf2"
server_key: "2d514f54025f0401d826cae296f13e8c"
server_key_iv: "7b5f3086cecc0e61d017d940101a2a3d"
server_key_version: "encryptionkey@0"
I20260621 06:51:50.177044 22053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "69f10dce80074ac691da28c98d82a6a4"
format_stamp: "Formatted at 2026-06-21 06:51:50 on dist-test-slave-gmf2"
server_key: "2d514f54025f0401d826cae296f13e8c"
server_key_iv: "7b5f3086cecc0e61d017d940101a2a3d"
server_key_version: "encryptionkey@0"
I20260621 06:51:50.181807 22053 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260621 06:51:50.184983 22068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:50.186229 22053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:51:50.186395 22053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "69f10dce80074ac691da28c98d82a6a4"
format_stamp: "Formatted at 2026-06-21 06:51:50 on dist-test-slave-gmf2"
server_key: "2d514f54025f0401d826cae296f13e8c"
server_key_iv: "7b5f3086cecc0e61d017d940101a2a3d"
server_key_version: "encryptionkey@0"
I20260621 06:51:50.186589 22053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:50.211470 22053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:50.215237 22053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:50.215523 22053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:50.235992 22053 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:45775
I20260621 06:51:50.236011 22129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:45775 every 8 connection(s)
I20260621 06:51:50.237288 22053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:51:50.239740 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 22053
I20260621 06:51:50.239977 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:51:50.240797 20963 external_mini_cluster.cc:1428] Setting key 077b657e28752e2bf20ce0c8bcdb14a6
I20260621 06:51:50.241417 22130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:50.254979 22130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4: Bootstrap starting.
Jun 21 06:51:50 dist-test-slave-gmf2 krb5kdc[22040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:50.257990 22130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:50.259004 22130 log.cc:824] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:50.261559 22130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4: No bootstrap required, opened a new log
I20260621 06:51:50.262953 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:50.242730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.20.120.254:45775)
0621 06:51:50.243578 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:50.243589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:50.243627 (+ 38us) server_negotiation.cc:407] Connection header received
0621 06:51:50.245720 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:50.245739 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:50.246127 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:50.246428 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:50.248071 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:50.249190 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:50.250911 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:50.251612 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:50.251760 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:50.256378 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:50.256385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:50.256401 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:50.256438 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:50.259338 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:50.259965 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:50.259971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:50.259979 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:50.260124 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:50.260483 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:50.260492 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:50.260494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:50.260920 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:50.261214 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:50.261622 (+ 408us) server_negotiation.cc:299] Negotiation successful
0621 06:51:50.262048 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":452,"threads_started":1}
I20260621 06:51:50.269333 22130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f10dce80074ac691da28c98d82a6a4" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45775 } }
I20260621 06:51:50.269654 22130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:50.269757 22130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f10dce80074ac691da28c98d82a6a4, State: Initialized, Role: FOLLOWER
I20260621 06:51:50.270272 22130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f10dce80074ac691da28c98d82a6a4" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45775 } }
I20260621 06:51:50.270507 22130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:51:50.270648 22130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:51:50.270812 22130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:50.273023 22130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f10dce80074ac691da28c98d82a6a4" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45775 } }
I20260621 06:51:50.273520 22130 leader_election.cc:302] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f10dce80074ac691da28c98d82a6a4; no voters:
I20260621 06:51:50.274176 22130 leader_election.cc:288] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:51:50.274271 22135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:51:50.274993 22135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [term 1 LEADER]: Becoming Leader. State: Replica: 69f10dce80074ac691da28c98d82a6a4, State: Running, Role: LEADER
I20260621 06:51:50.275516 22135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f10dce80074ac691da28c98d82a6a4" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45775 } }
I20260621 06:51:50.276178 22130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:51:50.278311 22137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69f10dce80074ac691da28c98d82a6a4. Latest consensus state: current_term: 1 leader_uuid: "69f10dce80074ac691da28c98d82a6a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f10dce80074ac691da28c98d82a6a4" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45775 } } }
I20260621 06:51:50.278679 22137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [sys.catalog]: This master's current role is: LEADER
I20260621 06:51:50.278462 22136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69f10dce80074ac691da28c98d82a6a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f10dce80074ac691da28c98d82a6a4" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45775 } } }
I20260621 06:51:50.278827 22136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4 [sys.catalog]: This master's current role is: LEADER
I20260621 06:51:50.283099 22142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:51:50.287842 22142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:51:50.296489 22142 catalog_manager.cc:1347] Generated new cluster ID: 1cc2462f68d54543accf4e50152afdc5
I20260621 06:51:50.296602 22142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:51:50.317154 22142 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:51:50.318495 22142 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:51:50.326994 22142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4: Generated new TSK 0
I20260621 06:51:50.327793 22142 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:51:50.384871 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45775
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:39809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:50.696166 22156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:50.696414 22156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:50.696580 22156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:50.703402 22156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:50.703488 22156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:50.703675 22156 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 06:51:50.703794 22156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:51:50.713007 22156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:39809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.120.254:45775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:50.714272 22156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:50.715514 22156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:50.727946 22164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:50.727977 22161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:50.728000 22162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:50.728881 22156 server_base.cc:1034] running on GCE node
I20260621 06:51:50.729414 22156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:50.730285 22156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:50.731503 22156 hybrid_clock.cc:648] HybridClock initialized: now 1782024710731432 us; error 53 us; skew 500 ppm
I20260621 06:51:50.731802 22156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:50 dist-test-slave-gmf2 krb5kdc[22040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024710, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:50.734715 22156 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:51:50.735898 22156 webserver.cc:460] Webserver started at http://127.20.120.193:41201/ using document root <none> and password file <none>
I20260621 06:51:50.736476 22156 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:50.736564 22156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:50.736800 22156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:50.739403 22156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "e7a685482bd6476480d0e356edb5f4b9"
format_stamp: "Formatted at 2026-06-21 06:51:50 on dist-test-slave-gmf2"
server_key: "7cd5b9c03ebc94811d590f4eef22b20d"
server_key_iv: "9c0b5e44d367a15c7a0bea7f1c3aec5f"
server_key_version: "encryptionkey@0"
I20260621 06:51:50.740092 22156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "e7a685482bd6476480d0e356edb5f4b9"
format_stamp: "Formatted at 2026-06-21 06:51:50 on dist-test-slave-gmf2"
server_key: "7cd5b9c03ebc94811d590f4eef22b20d"
server_key_iv: "9c0b5e44d367a15c7a0bea7f1c3aec5f"
server_key_version: "encryptionkey@0"
I20260621 06:51:50.744725 22156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 06:51:50.747856 22171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:50.749079 22156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 06:51:50.749228 22156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "e7a685482bd6476480d0e356edb5f4b9"
format_stamp: "Formatted at 2026-06-21 06:51:50 on dist-test-slave-gmf2"
server_key: "7cd5b9c03ebc94811d590f4eef22b20d"
server_key_iv: "9c0b5e44d367a15c7a0bea7f1c3aec5f"
server_key_version: "encryptionkey@0"
I20260621 06:51:50.749428 22156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:50.765601 22156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:50.768955 22156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:50.769192 22156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:50.771697 22156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:51:50.771773 22156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:50.771956 22156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:51:50.772039 22156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 06:51:50 dist-test-slave-gmf2 krb5kdc[22040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024710, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:50.793831 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:50.772044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:33553 (local address 127.20.120.254:45775)
0621 06:51:50.772462 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:50.772469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:50.773601 (+ 1132us) server_negotiation.cc:407] Connection header received
0621 06:51:50.775598 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:50.775604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:50.775703 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:50.775819 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:50.777967 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:50.779251 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:50.780852 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:50.781581 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:50.781730 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:50.786352 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:50.786362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:50.786369 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:50.786418 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:50.789603 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:50.790722 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:50.790728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:50.790734 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:50.790906 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:50.791557 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:50.791565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:50.791570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:50.791925 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:50.792095 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:50.792996 (+ 901us) server_negotiation.cc:299] Negotiation successful
0621 06:51:50.793292 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":203,"threads_started":1}
I20260621 06:51:50.794517 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:50.772589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45775 (local address 127.20.120.193:33553)
0621 06:51:50.773439 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:50.773483 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:50.775323 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:50.776037 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:50.776052 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:50.776819 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:50.777711 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:50.777743 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:50.779445 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:50.779452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:50.780708 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:50.780719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:50.781845 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:50.781853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:50.782126 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:50.782877 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:50.782939 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:50.786086 (+ 3147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:50.789748 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:50.789760 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:50.789779 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:50.790500 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:50.791018 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:50.791028 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:50.791031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:50.791368 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:50.792077 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:50.792085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:50.792447 (+ 362us) client_negotiation.cc:769] Sending connection context
0621 06:51:50.792821 (+ 374us) client_negotiation.cc:240] Negotiation successful
0621 06:51:50.793207 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":391,"threads_started":1}
I20260621 06:51:50.816855 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:50.803843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:59537 (local address 127.20.120.254:45775)
0621 06:51:50.804035 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:50.804043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:50.804570 (+ 527us) server_negotiation.cc:407] Connection header received
0621 06:51:50.804687 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:50.804696 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:50.804789 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:50.804912 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:50.805707 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:50.806904 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:50.808801 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:50.809218 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:50.809352 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:50.811531 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:50.811535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:50.811538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:50.811564 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:50.814431 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:50.815079 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:50.815086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:50.815090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:50.815202 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:50.815550 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:50.815556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:50.815562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:50.815814 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:50.815956 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:50.816316 (+ 360us) server_negotiation.cc:299] Negotiation successful
0621 06:51:50.816518 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 06:51:50.816946 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:50.803825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45775 (local address 127.20.120.193:59537)
0621 06:51:50.804468 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:50.804493 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:50.804596 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:50.804924 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:50.804927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:50.805326 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:50.805581 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:50.805589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:50.807066 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:50.807074 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:50.808662 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:50.808673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:50.809390 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:50.809395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:50.809551 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:50.810251 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:50.810282 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:50.811320 (+ 1038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:50.814570 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:50.814575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:50.814579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:50.814953 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:50.815321 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:50.815326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:50.815329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:50.815447 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:50.815925 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:50.815929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:50.816094 (+ 165us) client_negotiation.cc:769] Sending connection context
0621 06:51:50.816228 (+ 134us) client_negotiation.cc:240] Negotiation successful
0621 06:51:50.816514 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":356,"threads_started":1}
I20260621 06:51:50.824765 22156 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:40689
I20260621 06:51:50.824806 22291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:40689 every 8 connection(s)
I20260621 06:51:50.825878 22156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:51:50.827883 22292 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45775
I20260621 06:51:50.828056 22292 heartbeater.cc:464] Registering TS with master...
I20260621 06:51:50.828593 22292 heartbeater.cc:511] Master 127.20.120.254:45775 requested a full tablet report, sending...
I20260621 06:51:50.830474 22085 ts_manager.cc:194] Registered new tserver with Master: e7a685482bd6476480d0e356edb5f4b9 (127.20.120.193:40689)
I20260621 06:51:50.831883 22085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:33553
I20260621 06:51:50.834558 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 22156
I20260621 06:51:50.834770 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:51:50.835381 20963 external_mini_cluster.cc:1428] Setting key 56ff93ea1496beab37732564c5089827
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:51:50.871002 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45775
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:39809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:51.174238 22296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:51.174484 22296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:51.174681 22296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:51.181828 22296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:51.181967 22296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:51.182103 22296 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 06:51:51.182224 22296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:51:51.191190 22296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:39809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.120.254:45775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:51.192330 22296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:51.193578 22296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:51.205456 22304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:51.205607 22302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:51.206071 22296 server_base.cc:1034] running on GCE node
W20260621 06:51:51.206219 22301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:51.206791 22296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:51.207686 22296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:51.208933 22296 hybrid_clock.cc:648] HybridClock initialized: now 1782024711208882 us; error 35 us; skew 500 ppm
I20260621 06:51:51.209211 22296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:51 dist-test-slave-gmf2 krb5kdc[22040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024711, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:51.212180 22296 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:51:51.213488 22296 webserver.cc:460] Webserver started at http://127.20.120.194:44249/ using document root <none> and password file <none>
I20260621 06:51:51.214180 22296 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:51.214308 22296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:51.214572 22296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:51.217348 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "44953cb6854242a2b238cf26cc6e4715"
format_stamp: "Formatted at 2026-06-21 06:51:51 on dist-test-slave-gmf2"
server_key: "446d6c33ab9c44098d64ac634228cb44"
server_key_iv: "592b1b5765c3abce42caafdbd519a925"
server_key_version: "encryptionkey@0"
I20260621 06:51:51.218152 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "44953cb6854242a2b238cf26cc6e4715"
format_stamp: "Formatted at 2026-06-21 06:51:51 on dist-test-slave-gmf2"
server_key: "446d6c33ab9c44098d64ac634228cb44"
server_key_iv: "592b1b5765c3abce42caafdbd519a925"
server_key_version: "encryptionkey@0"
I20260621 06:51:51.223291 22296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 06:51:51.226579 22311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:51.227772 22296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:51:51.227939 22296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "44953cb6854242a2b238cf26cc6e4715"
format_stamp: "Formatted at 2026-06-21 06:51:51 on dist-test-slave-gmf2"
server_key: "446d6c33ab9c44098d64ac634228cb44"
server_key_iv: "592b1b5765c3abce42caafdbd519a925"
server_key_version: "encryptionkey@0"
I20260621 06:51:51.228137 22296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:51.249856 22296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:51.255121 22296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:51.255411 22296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:51.258435 22296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:51:51.258540 22296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:51.258735 22296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:51:51.258847 22296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 06:51:51 dist-test-slave-gmf2 krb5kdc[22040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024711, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:51.284430 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.258493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:47239 (local address 127.20.120.254:45775)
0621 06:51:51.258724 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:51.258732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:51.260220 (+ 1488us) server_negotiation.cc:407] Connection header received
0621 06:51:51.263208 (+ 2988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:51.263216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:51.263294 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:51.263376 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:51.265780 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.266559 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.270034 (+ 3475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.270471 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.270571 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.277845 (+ 7274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:51.277852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:51.277858 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:51.277894 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:51.281157 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:51.282178 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:51.282183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:51.282187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:51.282313 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:51.282759 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:51.282764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:51.282767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:51.283089 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:51.283250 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:51.283861 (+ 611us) server_negotiation.cc:299] Negotiation successful
0621 06:51:51.284093 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 06:51:51.285379 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.259046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45775 (local address 127.20.120.194:47239)
0621 06:51:51.260037 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:51.260083 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:51.262858 (+ 2775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:51.263681 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:51.263698 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:51.264584 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:51.265602 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.265635 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.266779 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.266787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.269837 (+ 3050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.269862 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.270692 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.270701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.271120 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.272054 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:51.272090 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:51.277595 (+ 5505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:51.281306 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:51.281314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:51.281327 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:51.282054 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:51.282415 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:51.282421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:51.282424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:51.282646 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:51.283214 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:51.283221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:51.283545 (+ 324us) client_negotiation.cc:769] Sending connection context
0621 06:51:51.283775 (+ 230us) client_negotiation.cc:240] Negotiation successful
0621 06:51:51.284266 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":443,"threads_started":1}
I20260621 06:51:51.303645 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.291051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:50911 (local address 127.20.120.254:45775)
0621 06:51:51.291195 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:51.291199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:51.291907 (+ 708us) server_negotiation.cc:407] Connection header received
0621 06:51:51.292100 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:51.292105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:51.292170 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:51.292265 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:51.293708 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.294673 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.296126 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.296592 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.296701 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.298529 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:51.298547 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:51.298552 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:51.298587 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:51.300989 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:51.301708 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:51.301716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:51.301721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:51.301854 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:51.302223 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:51.302230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:51.302234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:51.302642 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:51.302791 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:51.303154 (+ 363us) server_negotiation.cc:299] Negotiation successful
0621 06:51:51.303342 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 06:51:51.303788 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.291214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45775 (local address 127.20.120.194:50911)
0621 06:51:51.291806 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:51.291825 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:51.291968 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:51.292439 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:51.292445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:51.293099 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:51.293501 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.293513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.294879 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.294886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.295987 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.295996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.296751 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.296758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.296880 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.297566 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:51.297587 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:51.298305 (+ 718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:51.301147 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:51.301152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:51.301156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:51.301586 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:51.301978 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:51.301988 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:51.301991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:51.302103 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:51.302763 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:51.302768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:51.302931 (+ 163us) client_negotiation.cc:769] Sending connection context
0621 06:51:51.303076 (+ 145us) client_negotiation.cc:240] Negotiation successful
0621 06:51:51.303365 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":344,"threads_started":1}
I20260621 06:51:51.314592 22296 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:35541
I20260621 06:51:51.314628 22430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:35541 every 8 connection(s)
I20260621 06:51:51.315821 22296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:51:51.317267 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 22296
I20260621 06:51:51.317487 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:51:51.317731 22431 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45775
I20260621 06:51:51.318120 22431 heartbeater.cc:464] Registering TS with master...
I20260621 06:51:51.318125 20963 external_mini_cluster.cc:1428] Setting key 6e47461981b66e23a74e86496802e16e
I20260621 06:51:51.318797 22431 heartbeater.cc:511] Master 127.20.120.254:45775 requested a full tablet report, sending...
I20260621 06:51:51.320201 22085 ts_manager.cc:194] Registered new tserver with Master: 44953cb6854242a2b238cf26cc6e4715 (127.20.120.194:35541)
I20260621 06:51:51.320953 22085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:47239
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:51:51.356930 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45775
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:39809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:51.669307 22435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:51.669625 22435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:51.669775 22435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:51.676615 22435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:51.676752 22435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:51.676883 22435 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 06:51:51.677013 22435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:51:51.686223 22435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:39809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.120.254:45775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:51.687671 22435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:51.689019 22435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:51.706383 22441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:51.706411 22443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:51.706476 22440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:51.707372 22435 server_base.cc:1034] running on GCE node
I20260621 06:51:51.707908 22435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:51.708776 22435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:51.710040 22435 hybrid_clock.cc:648] HybridClock initialized: now 1782024711709967 us; error 50 us; skew 500 ppm
I20260621 06:51:51.710319 22435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:51 dist-test-slave-gmf2 krb5kdc[22040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024711, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:51.713462 22435 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:51:51.714819 22435 webserver.cc:460] Webserver started at http://127.20.120.195:33207/ using document root <none> and password file <none>
I20260621 06:51:51.715451 22435 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:51.715534 22435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:51.715795 22435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:51.718703 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "1db93e53edbd49d8927563dfc354b54b"
format_stamp: "Formatted at 2026-06-21 06:51:51 on dist-test-slave-gmf2"
server_key: "23c5e5e8c60e2b1e11172bc0c1b41366"
server_key_iv: "3ff5ee722b621735776a89279b9e8d0b"
server_key_version: "encryptionkey@0"
I20260621 06:51:51.719395 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "1db93e53edbd49d8927563dfc354b54b"
format_stamp: "Formatted at 2026-06-21 06:51:51 on dist-test-slave-gmf2"
server_key: "23c5e5e8c60e2b1e11172bc0c1b41366"
server_key_iv: "3ff5ee722b621735776a89279b9e8d0b"
server_key_version: "encryptionkey@0"
I20260621 06:51:51.724360 22435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:51:51.727509 22450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:51.728698 22435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 06:51:51.728847 22435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "1db93e53edbd49d8927563dfc354b54b"
format_stamp: "Formatted at 2026-06-21 06:51:51 on dist-test-slave-gmf2"
server_key: "23c5e5e8c60e2b1e11172bc0c1b41366"
server_key_iv: "3ff5ee722b621735776a89279b9e8d0b"
server_key_version: "encryptionkey@0"
I20260621 06:51:51.729043 22435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:51.750402 22435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:51.754902 22435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:51.755177 22435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:51.757675 22435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:51:51.757786 22435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:51.757974 22435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:51:51.758047 22435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 21 06:51:51 dist-test-slave-gmf2 krb5kdc[22040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024711, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:51.784727 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.758158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:53505 (local address 127.20.120.254:45775)
0621 06:51:51.758349 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:51.758356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:51.759711 (+ 1355us) server_negotiation.cc:407] Connection header received
0621 06:51:51.762645 (+ 2934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:51.762654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:51.762744 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:51.762858 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:51.765417 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.766414 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.769430 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.770180 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.770312 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.776051 (+ 5739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:51.776061 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:51.776067 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:51.776109 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:51.780128 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:51.781337 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:51.781346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:51.781351 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:51.781532 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:51.782192 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:51.782199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:51.782204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:51.782604 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:51.782752 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:51.783981 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0621 06:51:51.784275 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 06:51:51.786808 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.758671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45775 (local address 127.20.120.195:53505)
0621 06:51:51.759523 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:51.759581 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:51.762321 (+ 2740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:51.763193 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:51.763206 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:51.764130 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:51.765207 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.765243 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.766612 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.766620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.769251 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.769269 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.770466 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.770475 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.770862 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.771756 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:51.771797 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:51.775761 (+ 3964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:51.780315 (+ 4554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:51.780327 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:51.780345 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:51.781204 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:51.781690 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:51.781698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:51.781704 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:51.782044 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:51.782778 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:51.782790 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:51.783424 (+ 634us) client_negotiation.cc:769] Sending connection context
0621 06:51:51.783825 (+ 401us) client_negotiation.cc:240] Negotiation successful
0621 06:51:51.784608 (+ 783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":325,"threads_started":1}
I20260621 06:51:51.796561 22435 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:37099
I20260621 06:51:51.796690 22569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:37099 every 8 connection(s)
I20260621 06:51:51.797996 22435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:51:51.800140 22570 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45775
I20260621 06:51:51.800489 22570 heartbeater.cc:464] Registering TS with master...
I20260621 06:51:51.801105 22570 heartbeater.cc:511] Master 127.20.120.254:45775 requested a full tablet report, sending...
I20260621 06:51:51.802521 22085 ts_manager.cc:194] Registered new tserver with Master: 1db93e53edbd49d8927563dfc354b54b (127.20.120.195:37099)
I20260621 06:51:51.803469 22085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:53505
I20260621 06:51:51.804153 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 22435
I20260621 06:51:51.804337 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:51:51.804775 20963 external_mini_cluster.cc:1428] Setting key 09efcfc2ec2401343b3d01eaeb9e394c
I20260621 06:51:51.807440 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:51:51.809084 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.794707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:36201 (local address 127.20.120.254:45775)
0621 06:51:51.794853 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:51.794857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:51.795607 (+ 750us) server_negotiation.cc:407] Connection header received
0621 06:51:51.795798 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:51.795806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:51.795895 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:51.795999 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:51.797395 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.798222 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.800108 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.800743 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.800863 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.803095 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:51.803100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:51.803103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:51.803130 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:51.806419 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:51.807200 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:51.807207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:51.807211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:51.807333 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:51.807693 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:51.807698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:51.807702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:51.807953 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:51.808119 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:51.808521 (+ 402us) server_negotiation.cc:299] Negotiation successful
0621 06:51:51.808723 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 06:51:51.809131 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.794791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45775 (local address 127.20.120.195:36201)
0621 06:51:51.795505 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:51.795521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:51.795667 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:51.796253 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:51.796262 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:51.796933 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:51.797211 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.797267 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.798337 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.798345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.799933 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.799945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.801161 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.801167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.801346 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.801988 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:51.802013 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:51.802935 (+ 922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:51.806576 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:51.806580 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:51.806584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:51.807059 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:51.807439 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:51.807445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:51.807449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:51.807583 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:51.808070 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:51.808074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:51.808243 (+ 169us) client_negotiation.cc:769] Sending connection context
0621 06:51:51.808382 (+ 139us) client_negotiation.cc:240] Negotiation successful
0621 06:51:51.808700 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":434,"threads_started":1}
Jun 21 06:51:51 dist-test-slave-gmf2 krb5kdc[22040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260621 06:51:51.834394 22292 heartbeater.cc:503] Master 127.20.120.254:45775 was elected leader, sending a full tablet report...
Jun 21 06:51:51 dist-test-slave-gmf2 krb5kdc[22040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:51.852536 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.839275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.20.120.254:45775)
0621 06:51:51.839500 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:51.839507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:51.839700 (+ 193us) server_negotiation.cc:407] Connection header received
0621 06:51:51.839841 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:51.839845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:51.839919 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:51.840002 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:51.841346 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.842129 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.843369 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.843763 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.843851 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.847591 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:51.847596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:51.847600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:51.847631 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:51.850192 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:51.850776 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:51.850780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:51.850782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:51.850894 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:51.851254 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:51.851257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:51.851259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:51.851500 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:51.851679 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:51.851940 (+ 261us) server_negotiation.cc:299] Negotiation successful
0621 06:51:51.852106 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 06:51:51.856658 22085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:51:51.863221 22085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:51:51.911989 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.875139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.20.120.194:35541)
0621 06:51:51.904396 (+ 29257us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:51.904403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:51.904432 (+ 29us) server_negotiation.cc:407] Connection header received
0621 06:51:51.904532 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:51.904542 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:51.904845 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:51.905850 (+ 1005us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:51.906060 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.907430 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.909665 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.910928 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.911103 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.911269 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:51.911521 (+ 252us) server_negotiation.cc:299] Negotiation successful
0621 06:51:51.911702 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29107,"thread_start_us":28985,"threads_started":1}
I20260621 06:51:51.912076 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.874715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:35541 (local address 127.0.0.1:50426)
0621 06:51:51.901211 (+ 26496us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:51.901259 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:51.901451 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:51.905127 (+ 3676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:51.905136 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:51.905165 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:51.905639 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.905648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.907578 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.907583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.909507 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.909517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.911117 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.911123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.911301 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.911345 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 06:51:51.911474 (+ 129us) client_negotiation.cc:240] Negotiation successful
0621 06:51:51.911679 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26183,"mutex_wait_us":127,"spinlock_wait_cycles":896,"thread_start_us":30205,"threads_started":1}
I20260621 06:51:51.913584 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.876668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:40689 (local address 127.0.0.1:40242)
0621 06:51:51.905205 (+ 28537us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:51.905232 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:51.905355 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:51.906130 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:51.906134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:51.906160 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:51.906591 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.906605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.908151 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.908157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.910591 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.910605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.912689 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.912694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.912949 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.913002 (+ 53us) client_negotiation.cc:769] Sending connection context
0621 06:51:51.913083 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 06:51:51.913234 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28429,"mutex_wait_us":216,"thread_start_us":24988,"threads_started":1}
I20260621 06:51:51.913620 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.876940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.20.120.193:40689)
0621 06:51:51.904305 (+ 27365us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:51.904315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:51.905348 (+ 1033us) server_negotiation.cc:407] Connection header received
0621 06:51:51.905539 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:51.905548 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:51.905739 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:51.905936 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:51.906789 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.907997 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.910764 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.912196 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.912362 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.912512 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:51.913168 (+ 656us) server_negotiation.cc:299] Negotiation successful
0621 06:51:51.913328 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27167,"thread_start_us":27372,"threads_started":1}
I20260621 06:51:51.913687 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.875265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.20.120.195:37099)
0621 06:51:51.904994 (+ 29729us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:51.905005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:51.905039 (+ 34us) server_negotiation.cc:407] Connection header received
0621 06:51:51.905129 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:51.905139 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:51.905404 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:51.905618 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:51.906953 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.908301 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.910764 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.912196 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.912382 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.912512 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:51.913221 (+ 709us) server_negotiation.cc:299] Negotiation successful
0621 06:51:51.913380 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29539,"thread_start_us":29335,"threads_started":1}
I20260621 06:51:51.914376 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:51.874866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:37099 (local address 127.0.0.1:56324)
0621 06:51:51.903585 (+ 28719us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:51.903625 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:51.903744 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:51.905629 (+ 1885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:51.905632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:51.905653 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:51.906033 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.906045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.908430 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.908436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.910591 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:51.910605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:51.912678 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:51.912685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:51.912965 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:51.913006 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 06:51:51.913083 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 06:51:51.913984 (+ 901us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28561,"mutex_wait_us":805,"thread_start_us":28960,"threads_started":1}
I20260621 06:51:51.916251 22361 tablet_service.cc:1467] Processing CreateTablet for tablet f5de5b7c642042b581c3ec53fc5ffc8a (DEFAULT_TABLE table=test-table [id=ef7301f8fcb04e65a2a674004f9cb69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:51.917423 22361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5de5b7c642042b581c3ec53fc5ffc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:51.917397 22500 tablet_service.cc:1467] Processing CreateTablet for tablet f5de5b7c642042b581c3ec53fc5ffc8a (DEFAULT_TABLE table=test-table [id=ef7301f8fcb04e65a2a674004f9cb69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:51.917397 22222 tablet_service.cc:1467] Processing CreateTablet for tablet f5de5b7c642042b581c3ec53fc5ffc8a (DEFAULT_TABLE table=test-table [id=ef7301f8fcb04e65a2a674004f9cb69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:51.918681 22222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5de5b7c642042b581c3ec53fc5ffc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:51.918695 22500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5de5b7c642042b581c3ec53fc5ffc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:51.929749 22589 tablet_bootstrap.cc:492] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715: Bootstrap starting.
I20260621 06:51:51.933063 22589 tablet_bootstrap.cc:654] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:51.934396 22589 log.cc:824] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:51.934614 22591 tablet_bootstrap.cc:492] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9: Bootstrap starting.
I20260621 06:51:51.938594 22590 tablet_bootstrap.cc:492] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b: Bootstrap starting.
I20260621 06:51:51.939071 22591 tablet_bootstrap.cc:654] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:51.940944 22591 log.cc:824] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:51.941203 22590 tablet_bootstrap.cc:654] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:51.942358 22590 log.cc:824] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:51.943684 22591 tablet_bootstrap.cc:492] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9: No bootstrap required, opened a new log
I20260621 06:51:51.943990 22591 ts_tablet_manager.cc:1397] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260621 06:51:51.944706 22590 tablet_bootstrap.cc:492] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b: No bootstrap required, opened a new log
I20260621 06:51:51.944932 22590 ts_tablet_manager.cc:1397] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 06:51:51.947428 22589 tablet_bootstrap.cc:492] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715: No bootstrap required, opened a new log
I20260621 06:51:51.947726 22589 ts_tablet_manager.cc:1397] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20260621 06:51:51.952134 22590 raft_consensus.cc:348] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } }
I20260621 06:51:51.952603 22590 raft_consensus.cc:374] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:51.952710 22590 raft_consensus.cc:729] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db93e53edbd49d8927563dfc354b54b, State: Initialized, Role: FOLLOWER
I20260621 06:51:51.953383 22590 consensus_queue.cc:260] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } }
I20260621 06:51:51.954084 22591 raft_consensus.cc:348] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } }
I20260621 06:51:51.954540 22591 raft_consensus.cc:374] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:51.954627 22570 heartbeater.cc:503] Master 127.20.120.254:45775 was elected leader, sending a full tablet report...
I20260621 06:51:51.954636 22591 raft_consensus.cc:729] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a685482bd6476480d0e356edb5f4b9, State: Initialized, Role: FOLLOWER
I20260621 06:51:51.955183 22591 consensus_queue.cc:260] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } }
I20260621 06:51:51.956773 22591 ts_tablet_manager.cc:1428] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260621 06:51:51.957793 22590 ts_tablet_manager.cc:1428] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260621 06:51:51.957979 22589 raft_consensus.cc:348] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } }
I20260621 06:51:51.958341 22589 raft_consensus.cc:374] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:51.958453 22589 raft_consensus.cc:729] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44953cb6854242a2b238cf26cc6e4715, State: Initialized, Role: FOLLOWER
I20260621 06:51:51.959069 22589 consensus_queue.cc:260] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } }
I20260621 06:51:51.961426 22431 heartbeater.cc:503] Master 127.20.120.254:45775 was elected leader, sending a full tablet report...
I20260621 06:51:51.962003 22589 ts_tablet_manager.cc:1428] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
W20260621 06:51:52.050570 22571 tablet.cc:2369] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:51:52.068413 22432 tablet.cc:2369] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:51:52.078624 22293 tablet.cc:2369] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:51:52.096380 22597 raft_consensus.cc:482] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:51:52.096670 22597 raft_consensus.cc:504] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } }
I20260621 06:51:52.098508 22597 leader_election.cc:288] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1db93e53edbd49d8927563dfc354b54b (127.20.120.195:37099), e7a685482bd6476480d0e356edb5f4b9 (127.20.120.193:40689)
I20260621 06:51:52.105703 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.098939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:46201 (local address 127.20.120.195:37099)
0621 06:51:52.099174 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:52.099184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:52.100570 (+ 1386us) server_negotiation.cc:407] Connection header received
0621 06:51:52.100746 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:52.100750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:52.100802 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:52.100905 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:52.101502 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.102241 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.103999 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.104758 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.104943 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.105019 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:52.105288 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 06:51:52.105425 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 06:51:52.105705 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.099092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:60507 (local address 127.20.120.193:40689)
0621 06:51:52.099324 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:52.099332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:52.100493 (+ 1161us) server_negotiation.cc:407] Connection header received
0621 06:51:52.100667 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:52.100672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:52.100744 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:52.100831 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:52.101417 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.102221 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.104007 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.104764 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.104943 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.105025 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:52.105338 (+ 313us) server_negotiation.cc:299] Negotiation successful
0621 06:51:52.105455 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:51:52.105862 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.098893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:40689 (local address 127.20.120.194:60507)
0621 06:51:52.100395 (+ 1502us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:52.100410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:52.100524 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:52.100862 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:52.100866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:52.100895 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:52.101254 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.101262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.102347 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.102354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.103860 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.103871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.105014 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.105019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.105175 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.105196 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 06:51:52.105314 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 06:51:52.105427 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"mutex_wait_us":23,"thread_start_us":624,"threads_started":1}
I20260621 06:51:52.105921 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.098913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:37099 (local address 127.20.120.194:46201)
0621 06:51:52.100484 (+ 1571us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:52.100500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:52.100624 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:52.100915 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:52.100920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:52.100944 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:52.101367 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.101375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.102365 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.102371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.103863 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.103874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.104980 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.104987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.105139 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.105161 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 06:51:52.105265 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 06:51:52.105400 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1423,"mutex_wait_us":22,"thread_start_us":801,"threads_started":1}
I20260621 06:51:52.106920 22246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5de5b7c642042b581c3ec53fc5ffc8a" candidate_uuid: "44953cb6854242a2b238cf26cc6e4715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a685482bd6476480d0e356edb5f4b9" is_pre_election: true
I20260621 06:51:52.106973 22520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5de5b7c642042b581c3ec53fc5ffc8a" candidate_uuid: "44953cb6854242a2b238cf26cc6e4715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db93e53edbd49d8927563dfc354b54b" is_pre_election: true
I20260621 06:51:52.107407 22246 raft_consensus.cc:2454] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44953cb6854242a2b238cf26cc6e4715 in term 0.
I20260621 06:51:52.107439 22520 raft_consensus.cc:2454] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44953cb6854242a2b238cf26cc6e4715 in term 0.
I20260621 06:51:52.108093 22314 leader_election.cc:302] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1db93e53edbd49d8927563dfc354b54b, 44953cb6854242a2b238cf26cc6e4715, e7a685482bd6476480d0e356edb5f4b9; no voters:
I20260621 06:51:52.108537 22597 raft_consensus.cc:2781] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:51:52.108677 22597 raft_consensus.cc:482] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:51:52.108778 22597 raft_consensus.cc:3037] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:52.111294 22597 raft_consensus.cc:504] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } }
I20260621 06:51:52.111861 22597 leader_election.cc:288] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [CANDIDATE]: Term 1 election: Requested vote from peers 1db93e53edbd49d8927563dfc354b54b (127.20.120.195:37099), e7a685482bd6476480d0e356edb5f4b9 (127.20.120.193:40689)
I20260621 06:51:52.112272 22520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5de5b7c642042b581c3ec53fc5ffc8a" candidate_uuid: "44953cb6854242a2b238cf26cc6e4715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db93e53edbd49d8927563dfc354b54b"
I20260621 06:51:52.112311 22246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5de5b7c642042b581c3ec53fc5ffc8a" candidate_uuid: "44953cb6854242a2b238cf26cc6e4715" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a685482bd6476480d0e356edb5f4b9"
I20260621 06:51:52.112489 22520 raft_consensus.cc:3037] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:52.112493 22246 raft_consensus.cc:3037] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:52.115620 22246 raft_consensus.cc:2454] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44953cb6854242a2b238cf26cc6e4715 in term 1.
I20260621 06:51:52.115620 22520 raft_consensus.cc:2454] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44953cb6854242a2b238cf26cc6e4715 in term 1.
I20260621 06:51:52.116035 22314 leader_election.cc:302] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1db93e53edbd49d8927563dfc354b54b, 44953cb6854242a2b238cf26cc6e4715, e7a685482bd6476480d0e356edb5f4b9; no voters:
I20260621 06:51:52.116325 22597 raft_consensus.cc:2781] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:51:52.116750 22597 raft_consensus.cc:686] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 1 LEADER]: Becoming Leader. State: Replica: 44953cb6854242a2b238cf26cc6e4715, State: Running, Role: LEADER
I20260621 06:51:52.117205 22597 consensus_queue.cc:237] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } }
I20260621 06:51:52.121596 22085 catalog_manager.cc:5495] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44953cb6854242a2b238cf26cc6e4715 (127.20.120.194). New cstate: current_term: 1 leader_uuid: "44953cb6854242a2b238cf26cc6e4715" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20260621 06:51:52.149508 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.144515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.20.120.194:35541)
0621 06:51:52.144693 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:52.144699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:52.144758 (+ 59us) server_negotiation.cc:407] Connection header received
0621 06:51:52.145048 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:52.145053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:52.145131 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:52.145264 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:51:52.145601 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.146348 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.147618 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.147987 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.148124 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.148558 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:51:52.148959 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:51:52.149096 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:52.149221 (+ 125us) server_negotiation.cc:299] Negotiation successful
0621 06:51:52.149288 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 06:51:52.156765 22520 raft_consensus.cc:1261] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b [term 1 FOLLOWER]: Refusing update from remote peer 44953cb6854242a2b238cf26cc6e4715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:52.156788 22246 raft_consensus.cc:1261] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9 [term 1 FOLLOWER]: Refusing update from remote peer 44953cb6854242a2b238cf26cc6e4715: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:52.157581 22600 consensus_queue.cc:1035] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:52.157928 22597 consensus_queue.cc:1035] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:52.168172 22603 mvcc.cc:205] Tried to move back new op lower bound from 7299173220985798656 to 7299173220835631104. Current Snapshot: MvccSnapshot[applied={T|T < 7299173220985798656}]
I20260621 06:51:52.185350 22085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:51:52.185580 22085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:51:52.188954 22085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4: Sending DeleteTablet for 3 replicas of tablet f5de5b7c642042b581c3ec53fc5ffc8a
I20260621 06:51:52.190004 22361 tablet_service.cc:1514] Processing DeleteTablet for tablet f5de5b7c642042b581c3ec53fc5ffc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:52 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:50426
I20260621 06:51:52.190115 22500 tablet_service.cc:1514] Processing DeleteTablet for tablet f5de5b7c642042b581c3ec53fc5ffc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:52 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:56324
I20260621 06:51:52.190938 22612 tablet_replica.cc:330] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715: stopping tablet replica
I20260621 06:51:52.191345 22222 tablet_service.cc:1514] Processing DeleteTablet for tablet f5de5b7c642042b581c3ec53fc5ffc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:52 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:40242
I20260621 06:51:52.191424 22612 raft_consensus.cc:2229] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:51:52.192633 22612 raft_consensus.cc:2258] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:52.193563 22085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:51:52.194268 22085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:51:52.195037 22612 ts_tablet_manager.cc:1905] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:52.197506 22613 tablet_replica.cc:330] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b: stopping tablet replica
I20260621 06:51:52.197525 22614 tablet_replica.cc:330] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9: stopping tablet replica
I20260621 06:51:52.197898 22613 raft_consensus.cc:2229] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:51:52.198405 22613 raft_consensus.cc:2258] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:52.199880 22614 raft_consensus.cc:2229] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:51:52.202610 22361 tablet_service.cc:1467] Processing CreateTablet for tablet adbad8bd97b6468d8d03f0f9b065d61a (DEFAULT_TABLE table=test-table [id=3211a81ddb6b416494eebe9ff14745a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:52.202891 22500 tablet_service.cc:1467] Processing CreateTablet for tablet adbad8bd97b6468d8d03f0f9b065d61a (DEFAULT_TABLE table=test-table [id=3211a81ddb6b416494eebe9ff14745a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:52.203020 22361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbad8bd97b6468d8d03f0f9b065d61a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:52.203316 22500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbad8bd97b6468d8d03f0f9b065d61a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:52.203506 22614 raft_consensus.cc:2258] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:52.202178 22222 tablet_service.cc:1467] Processing CreateTablet for tablet adbad8bd97b6468d8d03f0f9b065d61a (DEFAULT_TABLE table=test-table [id=3211a81ddb6b416494eebe9ff14745a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:52.206312 22222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adbad8bd97b6468d8d03f0f9b065d61a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:52.208081 22614 ts_tablet_manager.cc:1905] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:52.208488 22613 ts_tablet_manager.cc:1905] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:52.209599 22590 tablet_bootstrap.cc:492] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b: Bootstrap starting.
I20260621 06:51:52.211256 22590 tablet_bootstrap.cc:654] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:52.217352 22613 ts_tablet_manager.cc:1918] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:51:52.218600 22591 tablet_bootstrap.cc:492] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9: Bootstrap starting.
I20260621 06:51:52.220131 22591 tablet_bootstrap.cc:654] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:52.223686 22612 ts_tablet_manager.cc:1918] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:51:52.225150 22589 tablet_bootstrap.cc:492] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715: Bootstrap starting.
I20260621 06:51:52.229841 22589 tablet_bootstrap.cc:654] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:52.233779 22612 log.cc:1196] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/f5de5b7c642042b581c3ec53fc5ffc8a
I20260621 06:51:52.235209 22612 ts_tablet_manager.cc:1939] T f5de5b7c642042b581c3ec53fc5ffc8a P 44953cb6854242a2b238cf26cc6e4715: Deleting consensus metadata
I20260621 06:51:52.235491 22591 tablet_bootstrap.cc:492] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9: No bootstrap required, opened a new log
I20260621 06:51:52.235700 22591 ts_tablet_manager.cc:1397] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.012s
I20260621 06:51:52.236562 22591 raft_consensus.cc:348] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.236802 22591 raft_consensus.cc:374] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:52.236922 22591 raft_consensus.cc:729] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a685482bd6476480d0e356edb5f4b9, State: Initialized, Role: FOLLOWER
I20260621 06:51:52.237200 22591 consensus_queue.cc:260] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.237779 22591 ts_tablet_manager.cc:1428] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 06:51:52.243912 22069 catalog_manager.cc:4841] TS 44953cb6854242a2b238cf26cc6e4715 (127.20.120.194:35541): tablet f5de5b7c642042b581c3ec53fc5ffc8a (table test-table [id=ef7301f8fcb04e65a2a674004f9cb69f]) successfully deleted
I20260621 06:51:52.244395 22614 ts_tablet_manager.cc:1918] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:51:52.244573 22614 log.cc:1196] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/f5de5b7c642042b581c3ec53fc5ffc8a
I20260621 06:51:52.245141 22614 ts_tablet_manager.cc:1939] T f5de5b7c642042b581c3ec53fc5ffc8a P e7a685482bd6476480d0e356edb5f4b9: Deleting consensus metadata
I20260621 06:51:52.246974 22071 catalog_manager.cc:4841] TS e7a685482bd6476480d0e356edb5f4b9 (127.20.120.193:40689): tablet f5de5b7c642042b581c3ec53fc5ffc8a (table test-table [id=ef7301f8fcb04e65a2a674004f9cb69f]) successfully deleted
I20260621 06:51:52.246994 22613 log.cc:1196] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/f5de5b7c642042b581c3ec53fc5ffc8a
I20260621 06:51:52.247542 22613 ts_tablet_manager.cc:1939] T f5de5b7c642042b581c3ec53fc5ffc8a P 1db93e53edbd49d8927563dfc354b54b: Deleting consensus metadata
I20260621 06:51:52.249224 22072 catalog_manager.cc:4841] TS 1db93e53edbd49d8927563dfc354b54b (127.20.120.195:37099): tablet f5de5b7c642042b581c3ec53fc5ffc8a (table test-table [id=ef7301f8fcb04e65a2a674004f9cb69f]) successfully deleted
I20260621 06:51:52.238664 22590 tablet_bootstrap.cc:492] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b: No bootstrap required, opened a new log
I20260621 06:51:52.249572 22590 ts_tablet_manager.cc:1397] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b: Time spent bootstrapping tablet: real 0.040s user 0.017s sys 0.007s
I20260621 06:51:52.250471 22590 raft_consensus.cc:348] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.250720 22590 raft_consensus.cc:374] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:52.250821 22590 raft_consensus.cc:729] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db93e53edbd49d8927563dfc354b54b, State: Initialized, Role: FOLLOWER
I20260621 06:51:52.251084 22590 consensus_queue.cc:260] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.252455 22590 ts_tablet_manager.cc:1428] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 06:51:52.254380 22589 tablet_bootstrap.cc:492] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715: No bootstrap required, opened a new log
I20260621 06:51:52.254555 22589 ts_tablet_manager.cc:1397] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.000s
I20260621 06:51:52.255853 22589 raft_consensus.cc:348] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.256095 22589 raft_consensus.cc:374] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:52.256204 22589 raft_consensus.cc:729] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44953cb6854242a2b238cf26cc6e4715, State: Initialized, Role: FOLLOWER
I20260621 06:51:52.256466 22589 consensus_queue.cc:260] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.257177 22589 ts_tablet_manager.cc:1428] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260621 06:51:52.626314 22596 raft_consensus.cc:482] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:51:52.626557 22596 raft_consensus.cc:504] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.627959 22596 leader_election.cc:288] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44953cb6854242a2b238cf26cc6e4715 (127.20.120.194:35541), 1db93e53edbd49d8927563dfc354b54b (127.20.120.195:37099)
I20260621 06:51:52.634840 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.628289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:42699 (local address 127.20.120.194:35541)
0621 06:51:52.628507 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:52.628516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:52.629260 (+ 744us) server_negotiation.cc:407] Connection header received
0621 06:51:52.629435 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:52.629443 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:52.629519 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:52.629642 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:52.630275 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.630999 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.633043 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.633866 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.634051 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.634100 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:52.634425 (+ 325us) server_negotiation.cc:299] Negotiation successful
0621 06:51:52.634568 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 06:51:52.634876 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.628142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:35541 (local address 127.20.120.193:42699)
0621 06:51:52.629129 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:52.629147 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:52.629296 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:52.629659 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:52.629664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:52.629687 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:52.630121 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.630135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.631132 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.631137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.632880 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.632895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.634057 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.634071 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.634260 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.634300 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 06:51:52.634398 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 06:51:52.634522 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":36,"thread_start_us":709,"threads_started":1}
I20260621 06:51:52.635141 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.628333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:52329 (local address 127.20.120.195:37099)
0621 06:51:52.628897 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:52.628902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:52.629472 (+ 570us) server_negotiation.cc:407] Connection header received
0621 06:51:52.629655 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:52.629660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:52.629728 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:52.629833 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:52.630444 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.631211 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.633291 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.634165 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.634325 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.634409 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:52.634751 (+ 342us) server_negotiation.cc:299] Negotiation successful
0621 06:51:52.634892 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":328,"threads_started":1}
I20260621 06:51:52.635593 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.628329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:37099 (local address 127.20.120.193:52329)
0621 06:51:52.629373 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:52.629389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:52.629498 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:52.629856 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:52.629859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:52.629878 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:52.630274 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.630284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.631343 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.631347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.633131 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.633144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.634398 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.634404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.634582 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.634619 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 06:51:52.634713 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 06:51:52.634829 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":34,"thread_start_us":714,"threads_started":1}
I20260621 06:51:52.635794 22384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbad8bd97b6468d8d03f0f9b065d61a" candidate_uuid: "e7a685482bd6476480d0e356edb5f4b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44953cb6854242a2b238cf26cc6e4715" is_pre_election: true
I20260621 06:51:52.636153 22384 raft_consensus.cc:2454] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7a685482bd6476480d0e356edb5f4b9 in term 0.
I20260621 06:51:52.636211 22520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbad8bd97b6468d8d03f0f9b065d61a" candidate_uuid: "e7a685482bd6476480d0e356edb5f4b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db93e53edbd49d8927563dfc354b54b" is_pre_election: true
I20260621 06:51:52.636415 22520 raft_consensus.cc:2454] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7a685482bd6476480d0e356edb5f4b9 in term 0.
I20260621 06:51:52.636740 22172 leader_election.cc:302] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1db93e53edbd49d8927563dfc354b54b, 44953cb6854242a2b238cf26cc6e4715, e7a685482bd6476480d0e356edb5f4b9; no voters:
I20260621 06:51:52.637104 22596 raft_consensus.cc:2781] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:51:52.637218 22596 raft_consensus.cc:482] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:51:52.637323 22596 raft_consensus.cc:3037] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:52.639389 22596 raft_consensus.cc:504] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.640327 22596 leader_election.cc:288] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [CANDIDATE]: Term 1 election: Requested vote from peers 44953cb6854242a2b238cf26cc6e4715 (127.20.120.194:35541), 1db93e53edbd49d8927563dfc354b54b (127.20.120.195:37099)
I20260621 06:51:52.640681 22384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbad8bd97b6468d8d03f0f9b065d61a" candidate_uuid: "e7a685482bd6476480d0e356edb5f4b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44953cb6854242a2b238cf26cc6e4715"
I20260621 06:51:52.640875 22384 raft_consensus.cc:3037] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:52.640827 22520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adbad8bd97b6468d8d03f0f9b065d61a" candidate_uuid: "e7a685482bd6476480d0e356edb5f4b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db93e53edbd49d8927563dfc354b54b"
I20260621 06:51:52.641042 22520 raft_consensus.cc:3037] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:52.642812 22384 raft_consensus.cc:2454] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7a685482bd6476480d0e356edb5f4b9 in term 1.
I20260621 06:51:52.642887 22520 raft_consensus.cc:2454] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7a685482bd6476480d0e356edb5f4b9 in term 1.
I20260621 06:51:52.643213 22172 leader_election.cc:302] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44953cb6854242a2b238cf26cc6e4715, e7a685482bd6476480d0e356edb5f4b9; no voters:
I20260621 06:51:52.643553 22596 raft_consensus.cc:2781] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:51:52.643934 22596 raft_consensus.cc:686] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 1 LEADER]: Becoming Leader. State: Replica: e7a685482bd6476480d0e356edb5f4b9, State: Running, Role: LEADER
I20260621 06:51:52.644309 22596 consensus_queue.cc:237] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.647393 22085 catalog_manager.cc:5495] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7a685482bd6476480d0e356edb5f4b9 (127.20.120.193). New cstate: current_term: 1 leader_uuid: "e7a685482bd6476480d0e356edb5f4b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } health_report { overall_health: UNKNOWN } } }
Jun 21 06:51:52 dist-test-slave-gmf2 krb5kdc[22040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024710, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:52.701092 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.687953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45775 (local address 127.0.0.1:41996)
0621 06:51:52.688754 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:52.688769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:52.688920 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:52.689489 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:52.689494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:52.690312 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:52.690487 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.690495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.691481 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.691484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.692451 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.692461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.693082 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.693085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.693199 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.694242 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:52.694258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:52.696338 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:52.699169 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:52.699178 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:52.699183 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:52.699533 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:52.699828 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:52.699831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:52.699833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:52.699958 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:52.700369 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:52.700374 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:52.700504 (+ 130us) client_negotiation.cc:769] Sending connection context
0621 06:51:52.700632 (+ 128us) client_negotiation.cc:240] Negotiation successful
0621 06:51:52.700812 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":517,"threads_started":1}
I20260621 06:51:52.701205 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.687971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.20.120.254:45775)
0621 06:51:52.688634 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:52.688638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:52.688878 (+ 240us) server_negotiation.cc:407] Connection header received
0621 06:51:52.689038 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:52.689041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:52.689100 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:52.689226 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:52.690674 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.691368 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.692562 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.692962 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.693041 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.696513 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:52.696518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:52.696521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:52.696552 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:52.699067 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:52.699636 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:52.699640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:52.699642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:52.699741 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:52.700056 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:52.700059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:52.700061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:52.700277 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:52.700369 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:52.700697 (+ 328us) server_negotiation.cc:299] Negotiation successful
0621 06:51:52.700912 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":554,"thread_start_us":391,"threads_started":1}
I20260621 06:51:52.705652 22085 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:41996:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 06:51:52.706605 22085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:51:52.712464 22222 tablet_service.cc:1467] Processing CreateTablet for tablet 97786a1a4b124bf1995ab47876d05658 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d4ae4111ed747d1a7f57650b3e81d4d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 06:51:52.712584 22361 tablet_service.cc:1467] Processing CreateTablet for tablet 97786a1a4b124bf1995ab47876d05658 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d4ae4111ed747d1a7f57650b3e81d4d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 06:51:52.712757 22500 tablet_service.cc:1467] Processing CreateTablet for tablet 97786a1a4b124bf1995ab47876d05658 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d4ae4111ed747d1a7f57650b3e81d4d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 06:51:52.712997 22222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97786a1a4b124bf1995ab47876d05658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:52.713093 22361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97786a1a4b124bf1995ab47876d05658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:52.713243 22500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97786a1a4b124bf1995ab47876d05658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:52.718255 22591 tablet_bootstrap.cc:492] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: Bootstrap starting.
I20260621 06:51:52.719696 22591 tablet_bootstrap.cc:654] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:52.720305 22589 tablet_bootstrap.cc:492] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715: Bootstrap starting.
I20260621 06:51:52.720315 22590 tablet_bootstrap.cc:492] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b: Bootstrap starting.
I20260621 06:51:52.721606 22590 tablet_bootstrap.cc:654] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:52.722429 22589 tablet_bootstrap.cc:654] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:52.723285 22590 tablet_bootstrap.cc:492] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b: No bootstrap required, opened a new log
I20260621 06:51:52.723428 22590 ts_tablet_manager.cc:1397] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 06:51:52.724078 22590 raft_consensus.cc:348] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.724345 22590 raft_consensus.cc:374] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:52.724439 22590 raft_consensus.cc:729] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db93e53edbd49d8927563dfc354b54b, State: Initialized, Role: FOLLOWER
I20260621 06:51:52.724742 22590 consensus_queue.cc:260] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.725244 22589 tablet_bootstrap.cc:492] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715: No bootstrap required, opened a new log
I20260621 06:51:52.725392 22589 ts_tablet_manager.cc:1397] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 06:51:52.725193 22595 tablet_replica.cc:439] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } }
I20260621 06:51:52.725517 22595 tablet_replica.cc:442] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:52.726084 22591 tablet_bootstrap.cc:492] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: No bootstrap required, opened a new log
I20260621 06:51:52.726223 22591 ts_tablet_manager.cc:1397] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260621 06:51:52.726127 22589 raft_consensus.cc:348] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.726318 22590 ts_tablet_manager.cc:1428] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 06:51:52.726341 22589 raft_consensus.cc:374] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:52.726405 22589 raft_consensus.cc:729] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44953cb6854242a2b238cf26cc6e4715, State: Initialized, Role: FOLLOWER
I20260621 06:51:52.726634 22589 consensus_queue.cc:260] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.727202 22589 ts_tablet_manager.cc:1428] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 06:51:52.727301 22591 raft_consensus.cc:348] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.727559 22591 raft_consensus.cc:374] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:52.727628 22591 raft_consensus.cc:729] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a685482bd6476480d0e356edb5f4b9, State: Initialized, Role: FOLLOWER
I20260621 06:51:52.727049 22600 tablet_replica.cc:439] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } }
I20260621 06:51:52.727875 22600 tablet_replica.cc:442] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:52.727857 22591 consensus_queue.cc:260] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.728245 22596 tablet_replica.cc:439] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } }
I20260621 06:51:52.728492 22596 tablet_replica.cc:442] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:52.728988 22591 ts_tablet_manager.cc:1428] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260621 06:51:52.817492 22596 raft_consensus.cc:482] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:51:52.817680 22596 raft_consensus.cc:504] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.818292 22596 leader_election.cc:288] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44953cb6854242a2b238cf26cc6e4715 (127.20.120.194:35541), 1db93e53edbd49d8927563dfc354b54b (127.20.120.195:37099)
I20260621 06:51:52.818744 22384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97786a1a4b124bf1995ab47876d05658" candidate_uuid: "e7a685482bd6476480d0e356edb5f4b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44953cb6854242a2b238cf26cc6e4715" is_pre_election: true
I20260621 06:51:52.818775 22520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97786a1a4b124bf1995ab47876d05658" candidate_uuid: "e7a685482bd6476480d0e356edb5f4b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db93e53edbd49d8927563dfc354b54b" is_pre_election: true
I20260621 06:51:52.819003 22520 raft_consensus.cc:2454] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7a685482bd6476480d0e356edb5f4b9 in term 0.
I20260621 06:51:52.818993 22384 raft_consensus.cc:2454] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7a685482bd6476480d0e356edb5f4b9 in term 0.
I20260621 06:51:52.819414 22175 leader_election.cc:302] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1db93e53edbd49d8927563dfc354b54b, e7a685482bd6476480d0e356edb5f4b9; no voters:
I20260621 06:51:52.819700 22596 raft_consensus.cc:2781] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:51:52.819803 22596 raft_consensus.cc:482] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:51:52.819967 22596 raft_consensus.cc:3037] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:52.821933 22596 raft_consensus.cc:504] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.822465 22596 leader_election.cc:288] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [CANDIDATE]: Term 1 election: Requested vote from peers 44953cb6854242a2b238cf26cc6e4715 (127.20.120.194:35541), 1db93e53edbd49d8927563dfc354b54b (127.20.120.195:37099)
I20260621 06:51:52.822825 22384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97786a1a4b124bf1995ab47876d05658" candidate_uuid: "e7a685482bd6476480d0e356edb5f4b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44953cb6854242a2b238cf26cc6e4715"
I20260621 06:51:52.822880 22520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97786a1a4b124bf1995ab47876d05658" candidate_uuid: "e7a685482bd6476480d0e356edb5f4b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db93e53edbd49d8927563dfc354b54b"
I20260621 06:51:52.823025 22384 raft_consensus.cc:3037] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:52.823045 22520 raft_consensus.cc:3037] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:52.826210 22520 raft_consensus.cc:2454] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7a685482bd6476480d0e356edb5f4b9 in term 1.
I20260621 06:51:52.826210 22384 raft_consensus.cc:2454] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7a685482bd6476480d0e356edb5f4b9 in term 1.
I20260621 06:51:52.826531 22175 leader_election.cc:302] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1db93e53edbd49d8927563dfc354b54b, e7a685482bd6476480d0e356edb5f4b9; no voters:
I20260621 06:51:52.826810 22596 raft_consensus.cc:2781] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:51:52.826975 22596 raft_consensus.cc:686] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [term 1 LEADER]: Becoming Leader. State: Replica: e7a685482bd6476480d0e356edb5f4b9, State: Running, Role: LEADER
I20260621 06:51:52.827205 22596 consensus_queue.cc:237] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } }
I20260621 06:51:52.828644 22621 tablet_replica.cc:439] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: TxnStatusTablet state changed. Reason: New leader e7a685482bd6476480d0e356edb5f4b9. Latest consensus state: current_term: 1 leader_uuid: "e7a685482bd6476480d0e356edb5f4b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } }
I20260621 06:51:52.828976 22621 tablet_replica.cc:442] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: This TxnStatusTablet replica's current role is: LEADER
I20260621 06:51:52.829599 22634 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 06:51:52.829831 22634 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 06:51:52.830188 22085 catalog_manager.cc:5495] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7a685482bd6476480d0e356edb5f4b9 (127.20.120.193). New cstate: current_term: 1 leader_uuid: "e7a685482bd6476480d0e356edb5f4b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } health_report { overall_health: UNKNOWN } } }
I20260621 06:51:52.857479 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.847657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:40689 (local address 127.0.0.1:40244)
0621 06:51:52.848018 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:52.848039 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:52.848183 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:52.850813 (+ 2630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:52.850820 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:52.850835 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 06:51:52.851023 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.851036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.852862 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.852868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.854543 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.854558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.855312 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.855319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.855537 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.855620 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 06:51:52.856614 (+ 994us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 06:51:52.856670 (+ 56us) client_negotiation.cc:769] Sending connection context
0621 06:51:52.856796 (+ 126us) client_negotiation.cc:240] Negotiation successful
0621 06:51:52.857030 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260621 06:51:52.857542 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.847634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.20.120.193:40689)
0621 06:51:52.850399 (+ 2765us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:52.850405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:52.850427 (+ 22us) server_negotiation.cc:407] Connection header received
0621 06:51:52.850510 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:52.850515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:52.850619 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:52.850762 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:51:52.851202 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.852716 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.854689 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.855162 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.855255 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.855816 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:51:52.856403 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:51:52.856599 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:52.856873 (+ 274us) server_negotiation.cc:299] Negotiation successful
0621 06:51:52.856974 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2611,"thread_start_us":2454,"threads_started":1}
I20260621 06:51:52.861901 22520 raft_consensus.cc:1261] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b [term 1 FOLLOWER]: Refusing update from remote peer e7a685482bd6476480d0e356edb5f4b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:52.862237 22384 raft_consensus.cc:1261] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715 [term 1 FOLLOWER]: Refusing update from remote peer e7a685482bd6476480d0e356edb5f4b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:52.862957 22621 consensus_queue.cc:1035] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:52.863287 22596 consensus_queue.cc:1035] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:52.872257 22600 tablet_replica.cc:439] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715: TxnStatusTablet state changed. Reason: New leader e7a685482bd6476480d0e356edb5f4b9. Latest consensus state: current_term: 1 leader_uuid: "e7a685482bd6476480d0e356edb5f4b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } }
I20260621 06:51:52.872536 22600 tablet_replica.cc:442] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:52.874406 22638 mvcc.cc:205] Tried to move back new op lower bound from 7299173223875428352 to 7299173223742193664. Current Snapshot: MvccSnapshot[applied={T|T < 7299173223875428352}]
I20260621 06:51:52.875020 22595 tablet_replica.cc:439] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b: TxnStatusTablet state changed. Reason: New leader e7a685482bd6476480d0e356edb5f4b9. Latest consensus state: current_term: 1 leader_uuid: "e7a685482bd6476480d0e356edb5f4b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } }
I20260621 06:51:52.875263 22595 tablet_replica.cc:442] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:52.877846 22600 tablet_replica.cc:439] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7a685482bd6476480d0e356edb5f4b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } }
I20260621 06:51:52.878159 22600 tablet_replica.cc:442] T 97786a1a4b124bf1995ab47876d05658 P 44953cb6854242a2b238cf26cc6e4715: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:52.880282 22595 tablet_replica.cc:439] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7a685482bd6476480d0e356edb5f4b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } }
I20260621 06:51:52.880487 22595 tablet_replica.cc:442] T 97786a1a4b124bf1995ab47876d05658 P 1db93e53edbd49d8927563dfc354b54b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:52.887337 22646 tablet_replica.cc:439] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7a685482bd6476480d0e356edb5f4b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } }
I20260621 06:51:52.887704 22646 tablet_replica.cc:442] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: This TxnStatusTablet replica's current role is: LEADER
I20260621 06:51:52.887933 22621 tablet_replica.cc:439] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7a685482bd6476480d0e356edb5f4b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a685482bd6476480d0e356edb5f4b9" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40689 } } peers { permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 } } peers { permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 } } }
I20260621 06:51:52.888149 22621 tablet_replica.cc:442] T 97786a1a4b124bf1995ab47876d05658 P e7a685482bd6476480d0e356edb5f4b9: This TxnStatusTablet replica's current role is: LEADER
I20260621 06:51:52.900169 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.894872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.20.120.193:40689)
0621 06:51:52.895139 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:52.895146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:52.895514 (+ 368us) server_negotiation.cc:407] Connection header received
0621 06:51:52.895708 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:52.895714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:52.895782 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:52.895876 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:51:52.896369 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.897219 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.898606 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.899006 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.899139 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.899500 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:51:52.899599 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:51:52.899691 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:52.899832 (+ 141us) server_negotiation.cc:299] Negotiation successful
0621 06:51:52.899905 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260621 06:51:52.916078 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.910288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:35619 (local address 127.20.120.193:40689)
0621 06:51:52.910441 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:52.910444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:52.911068 (+ 624us) server_negotiation.cc:407] Connection header received
0621 06:51:52.911236 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:52.911240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:52.911314 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:52.911413 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:51:52.911886 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.912608 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.914073 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.914791 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.914923 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.915305 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:51:52.915389 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:51:52.915497 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:52.915626 (+ 129us) server_negotiation.cc:299] Negotiation successful
0621 06:51:52.915714 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":44}
I20260621 06:51:52.916152 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:52.910413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:40689 (local address 127.20.120.193:35619)
0621 06:51:52.910968 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:52.910982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:52.911136 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:52.911416 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:52.911420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:52.911429 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 06:51:52.911741 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.911750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.912730 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.912737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.913889 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:52.913900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:52.914960 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:52.914966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:52.915094 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:52.915137 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 06:51:52.915495 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 06:51:52.915505 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 06:51:52.915606 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 06:51:52.915742 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":1,"thread_start_us":294,"threads_started":1}
I20260621 06:51:52.924284 22520 raft_consensus.cc:1261] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b [term 1 FOLLOWER]: Refusing update from remote peer e7a685482bd6476480d0e356edb5f4b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:52.924319 22384 raft_consensus.cc:1261] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715 [term 1 FOLLOWER]: Refusing update from remote peer e7a685482bd6476480d0e356edb5f4b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:52.924786 22621 consensus_queue.cc:1035] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44953cb6854242a2b238cf26cc6e4715" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:52.925014 22596 consensus_queue.cc:1035] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1db93e53edbd49d8927563dfc354b54b" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:52.985340 22085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:51:52.985607 22085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:51:52.988410 22085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69f10dce80074ac691da28c98d82a6a4: Sending DeleteTablet for 3 replicas of tablet adbad8bd97b6468d8d03f0f9b065d61a
I20260621 06:51:52.989264 22361 tablet_service.cc:1514] Processing DeleteTablet for tablet adbad8bd97b6468d8d03f0f9b065d61a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:52 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:50426
I20260621 06:51:52.989461 22500 tablet_service.cc:1514] Processing DeleteTablet for tablet adbad8bd97b6468d8d03f0f9b065d61a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:52 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:56324
I20260621 06:51:52.990037 22657 tablet_replica.cc:330] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715: stopping tablet replica
I20260621 06:51:52.990054 22221 tablet_service.cc:1514] Processing DeleteTablet for tablet adbad8bd97b6468d8d03f0f9b065d61a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:52 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:40242
I20260621 06:51:52.990259 22657 raft_consensus.cc:2229] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:51:52.990461 22657 raft_consensus.cc:2258] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:52.991856 22657 ts_tablet_manager.cc:1905] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:53.000629 22658 tablet_replica.cc:330] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b: stopping tablet replica
I20260621 06:51:53.000685 22657 ts_tablet_manager.cc:1918] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:51:53.000787 22657 log.cc:1196] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/adbad8bd97b6468d8d03f0f9b065d61a
I20260621 06:51:53.000870 22658 raft_consensus.cc:2229] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:51:53.001124 22658 raft_consensus.cc:2258] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:53.001233 22657 ts_tablet_manager.cc:1939] T adbad8bd97b6468d8d03f0f9b065d61a P 44953cb6854242a2b238cf26cc6e4715: Deleting consensus metadata
I20260621 06:51:53.002374 22659 tablet_replica.cc:330] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9: stopping tablet replica
I20260621 06:51:53.002336 22069 catalog_manager.cc:4841] TS 44953cb6854242a2b238cf26cc6e4715 (127.20.120.194:35541): tablet adbad8bd97b6468d8d03f0f9b065d61a (table test-table [id=3211a81ddb6b416494eebe9ff14745a9]) successfully deleted
I20260621 06:51:53.002629 22659 raft_consensus.cc:2229] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:51:53.003311 22659 raft_consensus.cc:2258] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:53.004315 22659 ts_tablet_manager.cc:1905] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:53.004700 22658 ts_tablet_manager.cc:1905] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:53.011786 22658 ts_tablet_manager.cc:1918] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:51:53.011912 22658 log.cc:1196] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/adbad8bd97b6468d8d03f0f9b065d61a
I20260621 06:51:53.012372 22658 ts_tablet_manager.cc:1939] T adbad8bd97b6468d8d03f0f9b065d61a P 1db93e53edbd49d8927563dfc354b54b: Deleting consensus metadata
I20260621 06:51:53.013289 22659 ts_tablet_manager.cc:1918] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:51:53.013365 22072 catalog_manager.cc:4841] TS 1db93e53edbd49d8927563dfc354b54b (127.20.120.195:37099): tablet adbad8bd97b6468d8d03f0f9b065d61a (table test-table [id=3211a81ddb6b416494eebe9ff14745a9]) successfully deleted
I20260621 06:51:53.013425 22659 log.cc:1196] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/adbad8bd97b6468d8d03f0f9b065d61a
I20260621 06:51:53.013881 22659 ts_tablet_manager.cc:1939] T adbad8bd97b6468d8d03f0f9b065d61a P e7a685482bd6476480d0e356edb5f4b9: Deleting consensus metadata
I20260621 06:51:53.015005 22071 catalog_manager.cc:4841] TS e7a685482bd6476480d0e356edb5f4b9 (127.20.120.193:40689): tablet adbad8bd97b6468d8d03f0f9b065d61a (table test-table [id=3211a81ddb6b416494eebe9ff14745a9]) successfully deleted
Jun 21 06:51:53 dist-test-slave-gmf2 krb5kdc[22040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:51:53.029243 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:53.004413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.20.120.193:40689)
0621 06:51:53.004636 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:53.004642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:53.011770 (+ 7128us) server_negotiation.cc:407] Connection header received
0621 06:51:53.012348 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:53.012353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:53.012427 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:53.012501 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:53.014839 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:53.015741 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:53.017479 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:53.018041 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:53.018197 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:53.022199 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:53.022214 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:53.022234 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:53.022276 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:53.026015 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:53.026679 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:53.026690 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:53.026698 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:53.026865 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:53.027199 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:53.027204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:53.027207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:53.027671 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:53.027993 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:53.028170 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 06:51:53.028409 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260621 06:51:53.030071 22287 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40248
I20260621 06:51:53.049275 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:53.033045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42012 (local address 127.20.120.254:45775)
0621 06:51:53.033249 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:53.033255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:53.033489 (+ 234us) server_negotiation.cc:407] Connection header received
0621 06:51:53.033733 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:53.033739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:53.033823 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:53.033946 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:53.035078 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:53.036214 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:53.037467 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:53.038145 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:53.038276 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:53.040347 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:53.040354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:53.040360 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:53.040404 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:53.046782 (+ 6378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:53.047400 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:53.047407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:53.047412 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:53.047553 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:53.047868 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:53.047873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:53.047877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:53.048224 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:53.048448 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:53.048723 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 06:51:53.048940 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260621 06:51:53.049877 22085 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42012
I20260621 06:51:53.053601 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 22156
I20260621 06:51:53.068210 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:53.054420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.20.120.193:40689)
0621 06:51:53.054734 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:53.054743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:53.055171 (+ 428us) server_negotiation.cc:407] Connection header received
0621 06:51:53.055288 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:53.055293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:53.055370 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:53.055475 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:53.056469 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:53.057660 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:53.058865 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:53.059522 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:53.059663 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:53.061704 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:53.061710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:53.061715 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:53.061759 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:53.065617 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:53.066244 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:53.066250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:53.066254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:53.066404 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:53.066719 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:53.066724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:53.066728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:53.067078 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:53.067331 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:53.067616 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 06:51:53.067841 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260621 06:51:53.068774 22287 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40258
W20260621 06:51:53.069327 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 22156: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:51:53.069504 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 22156
I20260621 06:51:53.088454 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 22296
Jun 21 06:51:53 dist-test-slave-gmf2 krb5kdc[22040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:51:53.103291 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:53.089256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.20.120.194:35541)
0621 06:51:53.089433 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:53.089438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:53.089498 (+ 60us) server_negotiation.cc:407] Connection header received
0621 06:51:53.089739 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:53.089745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:53.089875 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:53.090007 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:53.091182 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:53.091889 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:53.093080 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:53.093471 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:53.093597 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:53.097824 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:53.097858 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:53.097873 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:53.097901 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:53.100643 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:53.101292 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:53.101301 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:53.101305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:53.101439 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:53.101793 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:53.101798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:53.101801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:53.102256 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:53.102557 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:53.102774 (+ 217us) server_negotiation.cc:299] Negotiation successful
0621 06:51:53.102981 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260621 06:51:53.103917 22426 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50436
W20260621 06:51:53.104522 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 22296: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:51:53.104651 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 22296
I20260621 06:51:53.125032 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 22435
Jun 21 06:51:53 dist-test-slave-gmf2 krb5kdc[22040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:51:53.138268 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:53.125713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.20.120.195:37099)
0621 06:51:53.125870 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:53.125874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:53.125890 (+ 16us) server_negotiation.cc:407] Connection header received
0621 06:51:53.126051 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:53.126055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:53.126113 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:53.126189 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:53.127258 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:53.127893 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:53.128961 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:53.129335 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:53.129420 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:53.133402 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:53.133415 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:53.133431 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:53.133464 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:53.136068 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:53.136621 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:53.136628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:53.136630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:53.136750 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:53.137027 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:53.137030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:53.137032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:53.137357 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:53.137587 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:53.137764 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 06:51:53.137982 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260621 06:51:53.138913 22564 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56338
W20260621 06:51:53.139513 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 22435: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:51:53.139650 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 22435
I20260621 06:51:53.156440 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 22053
I20260621 06:51:53.157416 22125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:51:53.267174 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 22053
2026-06-21T06:51:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4677 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:51:53 dist-test-slave-gmf2 krb5kdc[22668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:51:53 dist-test-slave-gmf2 krb5kdc[22668](info): set up 2 sockets
Jun 21 06:51:53 dist-test-slave-gmf2 krb5kdc[22668](info): commencing operation
krb5kdc: starting...
W20260621 06:51:54.322088 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:51:54 dist-test-slave-gmf2 krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:51:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:51:54.495357 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44355
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:44355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:54.810633 22681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:54.810964 22681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:54.811153 22681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:54.818367 22681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:51:54.818516 22681 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 06:51:54.818619 22681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:54.818744 22681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:54.818842 22681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:51:54.818944 22681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:51:54.828469 22681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:44355
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:54.829743 22681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:54.831169 22681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:54.843420 22687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:54.843420 22689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:54.843725 22686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:54.844074 22681 server_base.cc:1034] running on GCE node
I20260621 06:51:54.844841 22681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:54.846246 22681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:54.847479 22681 hybrid_clock.cc:648] HybridClock initialized: now 1782024714847422 us; error 38 us; skew 500 ppm
I20260621 06:51:54.847800 22681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:54 dist-test-slave-gmf2 krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024714, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:54.851300 22681 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:51:54.852679 22681 webserver.cc:460] Webserver started at http://127.20.120.254:44703/ using document root <none> and password file <none>
I20260621 06:51:54.853399 22681 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:54.853528 22681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:54.853834 22681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:54.856698 22681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "f55de3a8a62a4c8dbdaeeb87f6eb411f"
format_stamp: "Formatted at 2026-06-21 06:51:54 on dist-test-slave-gmf2"
server_key: "8396ee0f596f54f7c5099a23269f0e02"
server_key_iv: "f39b1dcca5eeea765e79ec4cd2d9b9fc"
server_key_version: "encryptionkey@0"
I20260621 06:51:54.857466 22681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "f55de3a8a62a4c8dbdaeeb87f6eb411f"
format_stamp: "Formatted at 2026-06-21 06:51:54 on dist-test-slave-gmf2"
server_key: "8396ee0f596f54f7c5099a23269f0e02"
server_key_iv: "f39b1dcca5eeea765e79ec4cd2d9b9fc"
server_key_version: "encryptionkey@0"
I20260621 06:51:54.862548 22681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 06:51:54.865722 22696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:54.866982 22681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:51:54.867161 22681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "f55de3a8a62a4c8dbdaeeb87f6eb411f"
format_stamp: "Formatted at 2026-06-21 06:51:54 on dist-test-slave-gmf2"
server_key: "8396ee0f596f54f7c5099a23269f0e02"
server_key_iv: "f39b1dcca5eeea765e79ec4cd2d9b9fc"
server_key_version: "encryptionkey@0"
I20260621 06:51:54.867364 22681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:54.883473 22681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:54.887107 22681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:54.887363 22681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:54.908902 22681 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:44355
I20260621 06:51:54.908913 22757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:44355 every 8 connection(s)
I20260621 06:51:54.910341 22681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:51:54.910887 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 22681
I20260621 06:51:54.911098 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:51:54.911881 20963 external_mini_cluster.cc:1428] Setting key a9bcc42573457eddef23b0090cb52428
I20260621 06:51:54.914592 22758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 06:51:54 dist-test-slave-gmf2 krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:54.927834 22758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f: Bootstrap starting.
I20260621 06:51:54.932466 22758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:54.933991 22758 log.cc:824] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:54.937026 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:54.913743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.20.120.254:44355)
0621 06:51:54.914483 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:54.914497 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:54.914534 (+ 37us) server_negotiation.cc:407] Connection header received
0621 06:51:54.916082 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:54.916100 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:54.916432 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:54.916751 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:54.918500 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:54.919834 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:54.921557 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:54.922238 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:54.922364 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:54.927095 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:54.927123 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:54.927141 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:54.927190 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:54.931652 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:54.932609 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:54.932619 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:54.932629 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:54.932867 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:54.933328 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:54.933333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:54.933337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:54.934053 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:54.934351 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:54.935616 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0621 06:51:54.936078 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":368,"threads_started":1}
I20260621 06:51:54.937346 22758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f: No bootstrap required, opened a new log
I20260621 06:51:54.945416 22758 raft_consensus.cc:348] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55de3a8a62a4c8dbdaeeb87f6eb411f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44355 } }
I20260621 06:51:54.945779 22758 raft_consensus.cc:374] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:54.945880 22758 raft_consensus.cc:729] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f55de3a8a62a4c8dbdaeeb87f6eb411f, State: Initialized, Role: FOLLOWER
I20260621 06:51:54.946440 22758 consensus_queue.cc:260] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55de3a8a62a4c8dbdaeeb87f6eb411f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44355 } }
I20260621 06:51:54.946763 22758 raft_consensus.cc:388] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:51:54.946931 22758 raft_consensus.cc:482] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:51:54.947119 22758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:54.949160 22758 raft_consensus.cc:504] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55de3a8a62a4c8dbdaeeb87f6eb411f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44355 } }
I20260621 06:51:54.949561 22758 leader_election.cc:302] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f55de3a8a62a4c8dbdaeeb87f6eb411f; no voters:
I20260621 06:51:54.950132 22758 leader_election.cc:288] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:51:54.950253 22763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:51:54.950928 22763 raft_consensus.cc:686] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [term 1 LEADER]: Becoming Leader. State: Replica: f55de3a8a62a4c8dbdaeeb87f6eb411f, State: Running, Role: LEADER
I20260621 06:51:54.951428 22763 consensus_queue.cc:237] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55de3a8a62a4c8dbdaeeb87f6eb411f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44355 } }
I20260621 06:51:54.952138 22758 sys_catalog.cc:564] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:51:54.954116 22764 sys_catalog.cc:455] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f55de3a8a62a4c8dbdaeeb87f6eb411f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55de3a8a62a4c8dbdaeeb87f6eb411f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44355 } } }
I20260621 06:51:54.954489 22764 sys_catalog.cc:458] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [sys.catalog]: This master's current role is: LEADER
I20260621 06:51:54.954370 22765 sys_catalog.cc:455] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f55de3a8a62a4c8dbdaeeb87f6eb411f. Latest consensus state: current_term: 1 leader_uuid: "f55de3a8a62a4c8dbdaeeb87f6eb411f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f55de3a8a62a4c8dbdaeeb87f6eb411f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44355 } } }
I20260621 06:51:54.954624 22765 sys_catalog.cc:458] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f [sys.catalog]: This master's current role is: LEADER
I20260621 06:51:54.958496 22769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:51:54.963686 22769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:51:54.971666 22769 catalog_manager.cc:1347] Generated new cluster ID: 5aca1df59aee445e917e793bf0a7f263
I20260621 06:51:54.971792 22769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:51:54.990097 22769 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:51:54.992106 22769 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:51:55.000449 22769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f: Generated new TSK 0
I20260621 06:51:55.001472 22769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:51:55.053388 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44355
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:55.363819 22784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:55.364074 22784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:55.364180 22784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:55.371333 22784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:55.371438 22784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:55.371531 22784 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 06:51:55.371582 22784 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 06:51:55.371662 22784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:51:55.381102 22784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.120.254:44355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:55.382337 22784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:55.383531 22784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:55.396278 22789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:55.396243 22792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:55.396325 22790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:55.396987 22784 server_base.cc:1034] running on GCE node
I20260621 06:51:55.397485 22784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:55.398340 22784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:55.399554 22784 hybrid_clock.cc:648] HybridClock initialized: now 1782024715399503 us; error 62 us; skew 500 ppm
I20260621 06:51:55.399853 22784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:55 dist-test-slave-gmf2 krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024715, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:55.402904 22784 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:51:55.404217 22784 webserver.cc:460] Webserver started at http://127.20.120.193:45307/ using document root <none> and password file <none>
I20260621 06:51:55.404871 22784 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:55.405009 22784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:55.405301 22784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:55.408154 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "87c7c0e9db174615b17d3ecdf41edd2a"
format_stamp: "Formatted at 2026-06-21 06:51:55 on dist-test-slave-gmf2"
server_key: "d8a46e79c6c585bc140ff3c2bb7d3c6e"
server_key_iv: "15d52c0a0a93519f19601dba6da2d3b9"
server_key_version: "encryptionkey@0"
I20260621 06:51:55.408871 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "87c7c0e9db174615b17d3ecdf41edd2a"
format_stamp: "Formatted at 2026-06-21 06:51:55 on dist-test-slave-gmf2"
server_key: "d8a46e79c6c585bc140ff3c2bb7d3c6e"
server_key_iv: "15d52c0a0a93519f19601dba6da2d3b9"
server_key_version: "encryptionkey@0"
I20260621 06:51:55.413950 22784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 06:51:55.417301 22799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:55.418771 22784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 06:51:55.418949 22784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "87c7c0e9db174615b17d3ecdf41edd2a"
format_stamp: "Formatted at 2026-06-21 06:51:55 on dist-test-slave-gmf2"
server_key: "d8a46e79c6c585bc140ff3c2bb7d3c6e"
server_key_iv: "15d52c0a0a93519f19601dba6da2d3b9"
server_key_version: "encryptionkey@0"
I20260621 06:51:55.419162 22784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:55.453351 22784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:55.457324 22784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:55.457554 22784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:55.460105 22784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:51:55.460235 22784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:55.460378 22784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:51:55.460475 22784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 06:51:55 dist-test-slave-gmf2 krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024715, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:55.483862 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:55.460823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:55675 (local address 127.20.120.254:44355)
0621 06:51:55.461412 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:55.461435 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:55.462612 (+ 1177us) server_negotiation.cc:407] Connection header received
0621 06:51:55.464520 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:55.464526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:55.464611 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:55.464736 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:55.467016 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:55.468275 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:55.471017 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:55.471790 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:55.471962 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:55.476880 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:55.476885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:55.476888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:55.476919 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:55.479696 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:55.480590 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:55.480596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:55.480600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:55.480758 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:55.481272 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:55.481277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:55.481280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:55.481709 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:55.481865 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:55.482960 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0621 06:51:55.483272 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":368,"threads_started":1}
I20260621 06:51:55.484493 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:55.461519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44355 (local address 127.20.120.193:55675)
0621 06:51:55.462419 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:55.462468 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:55.464239 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:55.465052 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:55.465065 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:55.465888 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:55.466716 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:55.466734 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:55.468453 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:55.468461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:55.470786 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:55.470840 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:55.471999 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:55.472005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:55.472243 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:55.472994 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:55.473033 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:55.476647 (+ 3614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:55.479821 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:55.479832 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:55.479857 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:55.480412 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:55.480908 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:55.480912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:55.480917 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:55.481147 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:55.481845 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:55.481853 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:55.482408 (+ 555us) client_negotiation.cc:769] Sending connection context
0621 06:51:55.482821 (+ 413us) client_negotiation.cc:240] Negotiation successful
0621 06:51:55.483275 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":378,"threads_started":1}
I20260621 06:51:55.504933 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:55.490201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44355 (local address 127.20.120.193:58797)
0621 06:51:55.490813 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:55.490828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:55.490928 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:55.491292 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:55.491295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:55.491694 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:55.491954 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:55.491962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:55.493528 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:55.493536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:55.495256 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:55.495267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:55.496238 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:55.496247 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:55.496520 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:55.497248 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:55.497274 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:55.498737 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:55.501998 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:55.502006 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:55.502014 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:55.502714 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:55.503113 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:55.503118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:55.503131 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:55.503353 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:55.504140 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:55.504143 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:55.504290 (+ 147us) client_negotiation.cc:769] Sending connection context
0621 06:51:55.504390 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 06:51:55.504621 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":330,"threads_started":1}
I20260621 06:51:55.505378 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:55.490165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:58797 (local address 127.20.120.254:44355)
0621 06:51:55.490389 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:55.490398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:55.490900 (+ 502us) server_negotiation.cc:407] Connection header received
0621 06:51:55.491035 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:55.491061 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:55.491155 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:55.491282 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:55.492085 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:55.493378 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:55.495407 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:55.495994 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:55.496150 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:55.498932 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:55.498941 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:55.498947 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:55.498987 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:55.501846 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:55.502864 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:55.502868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:55.502870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:55.502988 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:55.503510 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:55.503519 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:55.503525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:55.504004 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:55.504182 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:55.504574 (+ 392us) server_negotiation.cc:299] Negotiation successful
0621 06:51:55.504888 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 06:51:55.513064 22784 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:42497
I20260621 06:51:55.513103 22919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:42497 every 8 connection(s)
I20260621 06:51:55.514340 22784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:51:55.516498 22920 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44355
I20260621 06:51:55.516724 22920 heartbeater.cc:464] Registering TS with master...
I20260621 06:51:55.517338 22920 heartbeater.cc:511] Master 127.20.120.254:44355 requested a full tablet report, sending...
I20260621 06:51:55.519133 22713 ts_manager.cc:194] Registered new tserver with Master: 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497)
I20260621 06:51:55.520129 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 22784
I20260621 06:51:55.520316 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:51:55.520588 22713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:55675
I20260621 06:51:55.520936 20963 external_mini_cluster.cc:1428] Setting key f28e4453ecefaf963e25d9e891571644
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:51:55.559154 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44355
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:55.875721 22924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:55.875999 22924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:55.876185 22924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:55.883385 22924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:55.883497 22924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:55.883637 22924 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 06:51:55.883699 22924 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 06:51:55.883807 22924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:51:55.893743 22924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.120.254:44355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:55.895103 22924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:55.896488 22924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:55.910468 22932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:55.910441 22930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:55.910900 22929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:55.911712 22924 server_base.cc:1034] running on GCE node
I20260621 06:51:55.912355 22924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:55.913374 22924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:55.914675 22924 hybrid_clock.cc:648] HybridClock initialized: now 1782024715914618 us; error 53 us; skew 500 ppm
I20260621 06:51:55.914992 22924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:55 dist-test-slave-gmf2 krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024715, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:55.918265 22924 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:51:55.919864 22924 webserver.cc:460] Webserver started at http://127.20.120.194:40613/ using document root <none> and password file <none>
I20260621 06:51:55.920540 22924 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:55.920686 22924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:55.921001 22924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:55.924078 22924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "13057e4f6ce347738b11c51a48d7a8a6"
format_stamp: "Formatted at 2026-06-21 06:51:55 on dist-test-slave-gmf2"
server_key: "b837cf2f2621cd367d24e10b1df9e507"
server_key_iv: "2fb1a2e959928a04d95eedb69bf30047"
server_key_version: "encryptionkey@0"
I20260621 06:51:55.924883 22924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "13057e4f6ce347738b11c51a48d7a8a6"
format_stamp: "Formatted at 2026-06-21 06:51:55 on dist-test-slave-gmf2"
server_key: "b837cf2f2621cd367d24e10b1df9e507"
server_key_iv: "2fb1a2e959928a04d95eedb69bf30047"
server_key_version: "encryptionkey@0"
I20260621 06:51:55.930240 22924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260621 06:51:55.933717 22939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:55.935184 22924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:51:55.935364 22924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "13057e4f6ce347738b11c51a48d7a8a6"
format_stamp: "Formatted at 2026-06-21 06:51:55 on dist-test-slave-gmf2"
server_key: "b837cf2f2621cd367d24e10b1df9e507"
server_key_iv: "2fb1a2e959928a04d95eedb69bf30047"
server_key_version: "encryptionkey@0"
I20260621 06:51:55.935575 22924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:55.986310 22924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:55.990550 22924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:55.990792 22924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:55.993273 22924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:51:55.993407 22924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:55.993533 22924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:51:55.993616 22924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 06:51:56 dist-test-slave-gmf2 krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024715, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:56.018102 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:55.994076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:38349 (local address 127.20.120.254:44355)
0621 06:51:55.995376 (+ 1300us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:55.995383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:55.995992 (+ 609us) server_negotiation.cc:407] Connection header received
0621 06:51:55.998370 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:55.998376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:55.998472 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:55.998578 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:56.001363 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.002322 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.004521 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.005291 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.005433 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.010555 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:56.010559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:56.010562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:56.010598 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:56.013708 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:56.014696 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:56.014700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:56.014703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:56.014821 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:56.015523 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:56.015530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:56.015534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:56.015963 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:56.016150 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:56.017211 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0621 06:51:56.017521 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150}
I20260621 06:51:56.018991 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:55.994854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44355 (local address 127.20.120.194:38349)
0621 06:51:55.995747 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:55.995796 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:55.997996 (+ 2200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:55.998931 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:55.998945 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:55.999940 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:56.001104 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.001140 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.002487 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.002491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.004348 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.004359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.005662 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.005668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.006046 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.006905 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:56.006950 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:56.010351 (+ 3401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:56.013847 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:56.013856 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:56.013876 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:56.014504 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:56.014984 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:56.014991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:56.014994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:56.015332 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:56.016120 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:56.016132 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:56.016717 (+ 585us) client_negotiation.cc:769] Sending connection context
0621 06:51:56.017076 (+ 359us) client_negotiation.cc:240] Negotiation successful
0621 06:51:56.017518 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":420,"threads_started":1}
I20260621 06:51:56.041538 22924 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:33395
I20260621 06:51:56.041581 23058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:33395 every 8 connection(s)
I20260621 06:51:56.042148 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.025648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:57031 (local address 127.20.120.254:44355)
0621 06:51:56.025795 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:56.025798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:56.026739 (+ 941us) server_negotiation.cc:407] Connection header received
0621 06:51:56.026999 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:56.027005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:56.027093 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:56.027217 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:56.029044 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.030020 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.032381 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.032760 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.032900 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.035455 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:56.035464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:56.035471 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:56.035515 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:56.039273 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:56.039961 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:56.039967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:56.039971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:56.040165 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:56.040604 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:56.040610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:56.040615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:56.040962 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:56.041138 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:56.041583 (+ 445us) server_negotiation.cc:299] Negotiation successful
0621 06:51:56.041796 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 06:51:56.042268 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.025740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44355 (local address 127.20.120.194:57031)
0621 06:51:56.026638 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:56.026657 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:56.026835 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:56.027428 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:56.027435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:56.028253 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:56.028865 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.028883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.030185 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.030192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.032243 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.032260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.032952 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.032959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.033124 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.034022 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:56.034047 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:56.035255 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:56.039388 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:56.039391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:56.039394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:56.039838 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:56.040295 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:56.040300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:56.040307 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:56.040436 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:56.041100 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:56.041107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:56.041302 (+ 195us) client_negotiation.cc:769] Sending connection context
0621 06:51:56.041488 (+ 186us) client_negotiation.cc:240] Negotiation successful
0621 06:51:56.041806 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":433,"threads_started":1}
I20260621 06:51:56.043249 22924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:51:56.045537 23059 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44355
I20260621 06:51:56.045768 23059 heartbeater.cc:464] Registering TS with master...
I20260621 06:51:56.046407 23059 heartbeater.cc:511] Master 127.20.120.254:44355 requested a full tablet report, sending...
I20260621 06:51:56.046993 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 22924
I20260621 06:51:56.047196 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:51:56.047645 22713 ts_manager.cc:194] Registered new tserver with Master: 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395)
I20260621 06:51:56.047672 20963 external_mini_cluster.cc:1428] Setting key 921de5050c0be71c570ecb2137d3cf2d
I20260621 06:51:56.048578 22713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:38349
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:51:56.085126 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44355
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:51:56.400405 23063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:51:56.400715 23063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:51:56.400878 23063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:51:56.407935 23063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:51:56.408079 23063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:51:56.408213 23063 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 06:51:56.408298 23063 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 06:51:56.408420 23063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:51:56.417682 23063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.120.254:44355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:51:56.418980 23063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:51:56.420300 23063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:51:56.433153 23068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:56.433182 23069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:51:56.433326 23071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:51:56.434007 23063 server_base.cc:1034] running on GCE node
I20260621 06:51:56.434656 23063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:51:56.435580 23063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:51:56.436803 23063 hybrid_clock.cc:648] HybridClock initialized: now 1782024716436768 us; error 50 us; skew 500 ppm
I20260621 06:51:56.437109 23063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:51:56 dist-test-slave-gmf2 krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024716, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:51:56.440238 23063 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:51:56.441747 23063 webserver.cc:460] Webserver started at http://127.20.120.195:33997/ using document root <none> and password file <none>
I20260621 06:51:56.442478 23063 fs_manager.cc:359] Metadata directory not provided
I20260621 06:51:56.442644 23063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:51:56.442963 23063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:51:56.445770 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "ed50517b59b94c54a902a4c4f4252d08"
format_stamp: "Formatted at 2026-06-21 06:51:56 on dist-test-slave-gmf2"
server_key: "fb0bbe1449a52ef4bba0fd63f34bc2e8"
server_key_iv: "98ca7b4c3e2592612cf3a9ae1f529668"
server_key_version: "encryptionkey@0"
I20260621 06:51:56.446593 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "ed50517b59b94c54a902a4c4f4252d08"
format_stamp: "Formatted at 2026-06-21 06:51:56 on dist-test-slave-gmf2"
server_key: "fb0bbe1449a52ef4bba0fd63f34bc2e8"
server_key_iv: "98ca7b4c3e2592612cf3a9ae1f529668"
server_key_version: "encryptionkey@0"
I20260621 06:51:56.451418 23063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 06:51:56.454840 23078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:56.456080 23063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:51:56.456250 23063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "ed50517b59b94c54a902a4c4f4252d08"
format_stamp: "Formatted at 2026-06-21 06:51:56 on dist-test-slave-gmf2"
server_key: "fb0bbe1449a52ef4bba0fd63f34bc2e8"
server_key_iv: "98ca7b4c3e2592612cf3a9ae1f529668"
server_key_version: "encryptionkey@0"
I20260621 06:51:56.456446 23063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:51:56.472368 23063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:51:56.476127 23063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:51:56.476419 23063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:51:56.479097 23063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:51:56.479225 23063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:51:56.479372 23063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:51:56.479458 23063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 06:51:56 dist-test-slave-gmf2 krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024716, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:56.502730 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.479435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:43547 (local address 127.20.120.254:44355)
0621 06:51:56.479611 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:56.479617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:56.481075 (+ 1458us) server_negotiation.cc:407] Connection header received
0621 06:51:56.483205 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:56.483213 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:56.483310 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:56.483434 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:56.485702 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.486812 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.489267 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.489921 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.490137 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.494935 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:56.494942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:56.494949 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:56.495007 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:56.498258 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:56.499198 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:56.499205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:56.499208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:56.499351 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:56.499997 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:56.500004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:56.500007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:56.500418 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:56.500579 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:56.502138 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0621 06:51:56.502353 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 06:51:56.503539 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.480012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44355 (local address 127.20.120.195:43547)
0621 06:51:56.480883 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:56.480936 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:56.482833 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:56.483701 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:56.483714 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:56.484543 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:56.485503 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.485534 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.486969 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.486977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.489053 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.489064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.490270 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.490279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.490467 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.491428 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:56.491449 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:56.494582 (+ 3133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:56.498392 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:56.498400 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:56.498414 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:56.499060 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:56.499476 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:56.499484 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:56.499488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:56.499861 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:56.500581 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:56.500603 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:56.501117 (+ 514us) client_negotiation.cc:769] Sending connection context
0621 06:51:56.501431 (+ 314us) client_negotiation.cc:240] Negotiation successful
0621 06:51:56.502073 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":356,"threads_started":1}
I20260621 06:51:56.523588 22920 heartbeater.cc:503] Master 127.20.120.254:44355 was elected leader, sending a full tablet report...
I20260621 06:51:56.530887 23063 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:34799
I20260621 06:51:56.530932 23197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:34799 every 8 connection(s)
I20260621 06:51:56.532593 23063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:51:56.533170 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.518312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44355 (local address 127.20.120.195:60489)
0621 06:51:56.519062 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:56.519084 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:56.519240 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:56.519599 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:56.519604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:56.520079 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:56.520423 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.520432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.522195 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.522200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.523785 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.523796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.524809 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.524819 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.525077 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.525817 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:56.525848 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:56.527272 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:56.530608 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:56.530617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:56.530621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:56.531055 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:56.531489 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:56.531493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:56.531495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:56.531613 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:56.532238 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:56.532241 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:56.532388 (+ 147us) client_negotiation.cc:769] Sending connection context
0621 06:51:56.532492 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 06:51:56.532785 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":349,"threads_started":1}
I20260621 06:51:56.533318 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.517544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:60489 (local address 127.20.120.254:44355)
0621 06:51:56.517805 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:56.517811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:56.519195 (+ 1384us) server_negotiation.cc:407] Connection header received
0621 06:51:56.519370 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:56.519373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:56.519446 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:56.519570 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:56.520662 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.521764 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.523932 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.524574 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.524709 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.527560 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:56.527568 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:56.527576 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:56.527618 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:56.530470 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:56.531226 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:56.531232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:56.531235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:56.531379 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:56.531748 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:56.531753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:56.531756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:56.532106 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:56.532257 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:56.532656 (+ 399us) server_negotiation.cc:299] Negotiation successful
0621 06:51:56.532879 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260621 06:51:56.534829 23198 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44355
I20260621 06:51:56.535058 23198 heartbeater.cc:464] Registering TS with master...
I20260621 06:51:56.535868 23198 heartbeater.cc:511] Master 127.20.120.254:44355 requested a full tablet report, sending...
I20260621 06:51:56.537454 22712 ts_manager.cc:194] Registered new tserver with Master: ed50517b59b94c54a902a4c4f4252d08 (127.20.120.195:34799)
I20260621 06:51:56.539865 22712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:43547
I20260621 06:51:56.542586 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 23063
I20260621 06:51:56.542761 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:51:56.543208 20963 external_mini_cluster.cc:1428] Setting key d121943e638f04de918ad749d961e8c2
I20260621 06:51:56.545718 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 06:51:56 dist-test-slave-gmf2 krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 06:51:56 dist-test-slave-gmf2 krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:56.590426 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.576960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.20.120.254:44355)
0621 06:51:56.577188 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:56.577194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:56.577406 (+ 212us) server_negotiation.cc:407] Connection header received
0621 06:51:56.577634 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:56.577638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:56.577700 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:56.577806 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:56.579176 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.579902 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.581135 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.581564 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.581701 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.585400 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:56.585405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:56.585408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:56.585440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:56.587903 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:56.588498 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:56.588505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:56.588509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:56.588645 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:56.589014 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:56.589020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:56.589024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:56.589295 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:56.589552 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:56.589836 (+ 284us) server_negotiation.cc:299] Negotiation successful
0621 06:51:56.590078 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 06:51:56.594451 22712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:51:56.601111 22712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:51:56.635113 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.612342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.20.120.195:34799)
0621 06:51:56.626502 (+ 14160us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:56.626513 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:56.626557 (+ 44us) server_negotiation.cc:407] Connection header received
0621 06:51:56.626676 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:56.626687 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:56.626975 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:56.627208 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:56.628349 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.630174 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.632831 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.633903 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.634163 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.634361 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:56.634590 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 06:51:56.634748 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13976,"thread_start_us":13807,"threads_started":1}
I20260621 06:51:56.635360 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.612246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:42497 (local address 127.0.0.1:46264)
0621 06:51:56.626817 (+ 14571us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:56.626837 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:56.626969 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:56.627726 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:56.627736 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:56.627761 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:56.628254 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.628269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.630668 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.630673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.632628 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.632645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.634495 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.634501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.634684 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.634713 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 06:51:56.634773 (+ 60us) client_negotiation.cc:240] Negotiation successful
0621 06:51:56.634909 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14445,"mutex_wait_us":60,"thread_start_us":14283,"threads_started":1}
I20260621 06:51:56.635782 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.612787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.20.120.193:42497)
0621 06:51:56.621204 (+ 8417us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:56.621217 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:56.627091 (+ 5874us) server_negotiation.cc:407] Connection header received
0621 06:51:56.627208 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:56.627216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:56.627487 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:56.628454 (+ 967us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:56.628676 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.630520 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.632811 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.634237 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.635078 (+ 841us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.635256 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:56.635323 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 06:51:56.635488 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8256,"thread_start_us":8132,"threads_started":1}
I20260621 06:51:56.636175 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.612676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:33395 (local address 127.0.0.1:35874)
0621 06:51:56.627397 (+ 14721us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:56.627414 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:56.627518 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:56.628353 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:56.628356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:56.628373 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:56.628610 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.628616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.630073 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.630078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.632982 (+ 2904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.632993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.634996 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.635001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.635179 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.635214 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 06:51:56.635640 (+ 426us) client_negotiation.cc:240] Negotiation successful
0621 06:51:56.635771 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14599,"mutex_wait_us":474,"thread_start_us":7774,"threads_started":1}
I20260621 06:51:56.636499 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.613288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35874 (local address 127.20.120.194:33395)
0621 06:51:56.626037 (+ 12749us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:56.626051 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:56.627645 (+ 1594us) server_negotiation.cc:407] Connection header received
0621 06:51:56.627746 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:56.627753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:56.627965 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:56.628147 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:56.628780 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.629892 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.633127 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.634498 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.634670 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.634851 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:56.636043 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0621 06:51:56.636179 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12499,"thread_start_us":12367,"threads_started":1}
I20260621 06:51:56.637401 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.612161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:34799 (local address 127.0.0.1:43414)
0621 06:51:56.622119 (+ 9958us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:56.622172 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:56.622397 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:56.627597 (+ 5200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:56.627607 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:56.627637 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:56.628118 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.628130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.630305 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.630310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.632664 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.632676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.634130 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.634138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.634337 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.634383 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 06:51:56.634544 (+ 161us) client_negotiation.cc:240] Negotiation successful
0621 06:51:56.634742 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8436,"mutex_wait_us":31,"thread_start_us":15810,"threads_started":1}
I20260621 06:51:56.640132 22989 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf5021577104b609fa7df218a2e872a (DEFAULT_TABLE table=test-table [id=fa16d7304da94e74af27c51dfb0731e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:56.640130 22850 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf5021577104b609fa7df218a2e872a (DEFAULT_TABLE table=test-table [id=fa16d7304da94e74af27c51dfb0731e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:56.640614 23128 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf5021577104b609fa7df218a2e872a (DEFAULT_TABLE table=test-table [id=fa16d7304da94e74af27c51dfb0731e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:56.641423 22850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf5021577104b609fa7df218a2e872a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:56.641428 22989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf5021577104b609fa7df218a2e872a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:56.641716 23128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf5021577104b609fa7df218a2e872a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:56.653347 23217 tablet_bootstrap.cc:492] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6: Bootstrap starting.
I20260621 06:51:56.656371 23217 tablet_bootstrap.cc:654] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:56.657541 23217 log.cc:824] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:56.660111 23219 tablet_bootstrap.cc:492] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08: Bootstrap starting.
I20260621 06:51:56.660698 23217 tablet_bootstrap.cc:492] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6: No bootstrap required, opened a new log
I20260621 06:51:56.660964 23217 ts_tablet_manager.cc:1397] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260621 06:51:56.661115 23218 tablet_bootstrap.cc:492] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a: Bootstrap starting.
I20260621 06:51:56.663795 23219 tablet_bootstrap.cc:654] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:56.663841 23218 tablet_bootstrap.cc:654] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:56.664774 23218 log.cc:824] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:56.664932 23219 log.cc:824] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08: Log is configured to *not* fsync() on all Append() calls
I20260621 06:51:56.667376 23219 tablet_bootstrap.cc:492] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08: No bootstrap required, opened a new log
I20260621 06:51:56.667646 23218 tablet_bootstrap.cc:492] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a: No bootstrap required, opened a new log
I20260621 06:51:56.667670 23219 ts_tablet_manager.cc:1397] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260621 06:51:56.667881 23218 ts_tablet_manager.cc:1397] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260621 06:51:56.671607 23217 raft_consensus.cc:348] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } }
I20260621 06:51:56.672052 23217 raft_consensus.cc:374] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:56.672164 23217 raft_consensus.cc:729] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13057e4f6ce347738b11c51a48d7a8a6, State: Initialized, Role: FOLLOWER
I20260621 06:51:56.672744 23217 consensus_queue.cc:260] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } }
I20260621 06:51:56.675545 23219 raft_consensus.cc:348] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } }
I20260621 06:51:56.675853 23219 raft_consensus.cc:374] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:56.675967 23219 raft_consensus.cc:729] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed50517b59b94c54a902a4c4f4252d08, State: Initialized, Role: FOLLOWER
I20260621 06:51:56.676496 23219 consensus_queue.cc:260] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } }
I20260621 06:51:56.679925 23059 heartbeater.cc:503] Master 127.20.120.254:44355 was elected leader, sending a full tablet report...
I20260621 06:51:56.679762 23218 raft_consensus.cc:348] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } }
I20260621 06:51:56.680190 23218 raft_consensus.cc:374] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:56.680310 23218 raft_consensus.cc:729] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c7c0e9db174615b17d3ecdf41edd2a, State: Initialized, Role: FOLLOWER
I20260621 06:51:56.680903 23217 ts_tablet_manager.cc:1428] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
I20260621 06:51:56.680981 23218 consensus_queue.cc:260] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } }
I20260621 06:51:56.682562 23218 ts_tablet_manager.cc:1428] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260621 06:51:56.683332 23198 heartbeater.cc:503] Master 127.20.120.254:44355 was elected leader, sending a full tablet report...
I20260621 06:51:56.684006 23219 ts_tablet_manager.cc:1428] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
W20260621 06:51:56.766535 22921 tablet.cc:2369] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:51:56.788785 23199 tablet.cc:2369] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:51:56.795222 23060 tablet.cc:2369] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:51:56.834403 23223 raft_consensus.cc:482] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:51:56.834663 23223 raft_consensus.cc:504] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } }
I20260621 06:51:56.836663 23223 leader_election.cc:288] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed50517b59b94c54a902a4c4f4252d08 (127.20.120.195:34799), 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497)
I20260621 06:51:56.844736 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.837064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:55565 (local address 127.20.120.195:34799)
0621 06:51:56.837282 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:56.837290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:56.838099 (+ 809us) server_negotiation.cc:407] Connection header received
0621 06:51:56.838284 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:56.838291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:56.838363 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:56.838452 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:56.838998 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.839974 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.842423 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.843646 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.843800 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.843867 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:56.844352 (+ 485us) server_negotiation.cc:299] Negotiation successful
0621 06:51:56.844493 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:51:56.844732 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.837317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:59573 (local address 127.20.120.193:42497)
0621 06:51:56.837554 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:56.837563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:56.837775 (+ 212us) server_negotiation.cc:407] Connection header received
0621 06:51:56.838024 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:56.838028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:56.838087 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:56.838163 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:56.838664 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.839754 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.842421 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.843646 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.843798 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.843867 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:56.844352 (+ 485us) server_negotiation.cc:299] Negotiation successful
0621 06:51:56.844489 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 06:51:56.845033 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.836980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:34799 (local address 127.20.120.194:55565)
0621 06:51:56.837978 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:56.838002 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:56.838134 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:56.838504 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:56.838510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:56.838555 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:56.838870 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.838877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.840090 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.840095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.842254 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.842277 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.843906 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.843912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.844139 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.844229 (+ 90us) client_negotiation.cc:769] Sending connection context
0621 06:51:56.844317 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 06:51:56.844547 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":107,"thread_start_us":637,"threads_started":1}
I20260621 06:51:56.845057 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.836944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:42497 (local address 127.20.120.194:59573)
0621 06:51:56.837647 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:56.837672 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:56.837840 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:56.838185 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:56.838191 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:56.838218 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:56.838534 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.838545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.839862 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.839865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.842258 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.842267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.843910 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.843914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.844144 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.844226 (+ 82us) client_negotiation.cc:769] Sending connection context
0621 06:51:56.844317 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 06:51:56.844547 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":151,"thread_start_us":348,"threads_started":1}
I20260621 06:51:56.846046 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf5021577104b609fa7df218a2e872a" candidate_uuid: "13057e4f6ce347738b11c51a48d7a8a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" is_pre_election: true
I20260621 06:51:56.846053 23148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf5021577104b609fa7df218a2e872a" candidate_uuid: "13057e4f6ce347738b11c51a48d7a8a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed50517b59b94c54a902a4c4f4252d08" is_pre_election: true
I20260621 06:51:56.846527 22875 raft_consensus.cc:2454] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13057e4f6ce347738b11c51a48d7a8a6 in term 0.
I20260621 06:51:56.846534 23148 raft_consensus.cc:2454] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13057e4f6ce347738b11c51a48d7a8a6 in term 0.
I20260621 06:51:56.847185 22940 leader_election.cc:302] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13057e4f6ce347738b11c51a48d7a8a6, ed50517b59b94c54a902a4c4f4252d08; no voters:
I20260621 06:51:56.847597 23223 raft_consensus.cc:2781] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:51:56.847736 23223 raft_consensus.cc:482] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:51:56.847841 23223 raft_consensus.cc:3037] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:56.849893 23223 raft_consensus.cc:504] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } }
I20260621 06:51:56.850605 23223 leader_election.cc:288] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [CANDIDATE]: Term 1 election: Requested vote from peers ed50517b59b94c54a902a4c4f4252d08 (127.20.120.195:34799), 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497)
I20260621 06:51:56.850947 23148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf5021577104b609fa7df218a2e872a" candidate_uuid: "13057e4f6ce347738b11c51a48d7a8a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed50517b59b94c54a902a4c4f4252d08"
I20260621 06:51:56.851076 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf5021577104b609fa7df218a2e872a" candidate_uuid: "13057e4f6ce347738b11c51a48d7a8a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c7c0e9db174615b17d3ecdf41edd2a"
I20260621 06:51:56.851166 23148 raft_consensus.cc:3037] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:56.851269 22875 raft_consensus.cc:3037] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:56.854430 22875 raft_consensus.cc:2454] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13057e4f6ce347738b11c51a48d7a8a6 in term 1.
I20260621 06:51:56.854442 23148 raft_consensus.cc:2454] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13057e4f6ce347738b11c51a48d7a8a6 in term 1.
I20260621 06:51:56.854827 22941 leader_election.cc:302] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13057e4f6ce347738b11c51a48d7a8a6, 87c7c0e9db174615b17d3ecdf41edd2a; no voters:
I20260621 06:51:56.855124 23223 raft_consensus.cc:2781] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:51:56.855556 23223 raft_consensus.cc:686] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 LEADER]: Becoming Leader. State: Replica: 13057e4f6ce347738b11c51a48d7a8a6, State: Running, Role: LEADER
I20260621 06:51:56.856225 23223 consensus_queue.cc:237] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } }
I20260621 06:51:56.860577 22713 catalog_manager.cc:5495] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194). New cstate: current_term: 1 leader_uuid: "13057e4f6ce347738b11c51a48d7a8a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } health_report { overall_health: HEALTHY } } }
Jun 21 06:51:56 dist-test-slave-gmf2 krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024714, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:56.957047 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.944122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55524 (local address 127.20.120.254:44355)
0621 06:51:56.944295 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:56.944300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:56.944785 (+ 485us) server_negotiation.cc:407] Connection header received
0621 06:51:56.944974 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:56.944983 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:56.945043 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:56.945149 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:56.946261 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.946971 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.948207 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.948611 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.948751 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.952048 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:56.952053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:56.952056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:56.952093 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:56.954577 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:56.955269 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:56.955273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:56.955275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:56.955379 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:56.955714 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:56.955717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:56.955719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:56.955995 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:56.956111 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:56.956498 (+ 387us) server_negotiation.cc:299] Negotiation successful
0621 06:51:56.956723 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":72}
I20260621 06:51:56.957100 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:56.944157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44355 (local address 127.0.0.1:55524)
0621 06:51:56.944680 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:56.944695 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:56.944855 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:56.945175 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:56.945179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:56.945854 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:51:56.946126 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.946135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.947086 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.947092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.948065 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:56.948076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:56.948787 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:56.948793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:56.948967 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:56.949639 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:51:56.949657 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:51:56.951831 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:51:56.954701 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:56.954711 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:56.954717 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:56.955167 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:56.955473 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:51:56.955477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:51:56.955479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:51:56.955622 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:51:56.956109 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:51:56.956114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:51:56.956266 (+ 152us) client_negotiation.cc:769] Sending connection context
0621 06:51:56.956427 (+ 161us) client_negotiation.cc:240] Negotiation successful
0621 06:51:56.956616 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":297,"threads_started":1}
I20260621 06:51:56.962061 22713 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:55524:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 06:51:56.962984 22713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:51:56.969431 22850 tablet_service.cc:1467] Processing CreateTablet for tablet 69c252660ba145d3ba66c152c1175926 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ff71c17b6754b538e081ab7f596c6a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 06:51:56.969769 22989 tablet_service.cc:1467] Processing CreateTablet for tablet 69c252660ba145d3ba66c152c1175926 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ff71c17b6754b538e081ab7f596c6a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 06:51:56.970407 22850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69c252660ba145d3ba66c152c1175926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:56.969972 23128 tablet_service.cc:1467] Processing CreateTablet for tablet 69c252660ba145d3ba66c152c1175926 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ff71c17b6754b538e081ab7f596c6a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 06:51:56.970723 22989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69c252660ba145d3ba66c152c1175926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:56.970749 23128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69c252660ba145d3ba66c152c1175926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:56.977039 23217 tablet_bootstrap.cc:492] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6: Bootstrap starting.
I20260621 06:51:56.977298 23219 tablet_bootstrap.cc:492] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: Bootstrap starting.
I20260621 06:51:56.978221 23218 tablet_bootstrap.cc:492] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a: Bootstrap starting.
I20260621 06:51:56.978627 23219 tablet_bootstrap.cc:654] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:56.978711 23217 tablet_bootstrap.cc:654] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:56.979976 23218 tablet_bootstrap.cc:654] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:56.980531 23219 tablet_bootstrap.cc:492] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: No bootstrap required, opened a new log
I20260621 06:51:56.980672 23219 ts_tablet_manager.cc:1397] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260621 06:51:56.981498 23217 tablet_bootstrap.cc:492] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6: No bootstrap required, opened a new log
I20260621 06:51:56.981432 23219 raft_consensus.cc:348] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:56.981643 23217 ts_tablet_manager.cc:1397] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 06:51:56.981686 23219 raft_consensus.cc:374] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:56.981751 23219 raft_consensus.cc:729] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed50517b59b94c54a902a4c4f4252d08, State: Initialized, Role: FOLLOWER
I20260621 06:51:56.981983 23219 consensus_queue.cc:260] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:56.982312 23218 tablet_bootstrap.cc:492] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a: No bootstrap required, opened a new log
I20260621 06:51:56.982468 23218 ts_tablet_manager.cc:1397] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 06:51:56.982532 23219 ts_tablet_manager.cc:1428] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 06:51:56.982372 23224 tablet_replica.cc:439] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } }
I20260621 06:51:56.982493 23217 raft_consensus.cc:348] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:56.982725 23224 tablet_replica.cc:442] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:56.982750 23217 raft_consensus.cc:374] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:56.982836 23217 raft_consensus.cc:729] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13057e4f6ce347738b11c51a48d7a8a6, State: Initialized, Role: FOLLOWER
I20260621 06:51:56.983084 23217 consensus_queue.cc:260] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:56.983259 23218 raft_consensus.cc:348] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:56.983672 23218 raft_consensus.cc:374] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:56.983553 23223 tablet_replica.cc:439] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } }
I20260621 06:51:56.983846 23223 tablet_replica.cc:442] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:56.983785 23218 raft_consensus.cc:729] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c7c0e9db174615b17d3ecdf41edd2a, State: Initialized, Role: FOLLOWER
I20260621 06:51:56.984162 23218 consensus_queue.cc:260] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:56.984691 23225 tablet_replica.cc:439] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } }
I20260621 06:51:56.985095 23225 tablet_replica.cc:442] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:56.985419 23217 ts_tablet_manager.cc:1428] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260621 06:51:56.985651 23218 ts_tablet_manager.cc:1428] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 06:51:57.130522 23224 raft_consensus.cc:482] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:51:57.130766 23224 raft_consensus.cc:504] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:57.132320 23224 leader_election.cc:288] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497), 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395)
I20260621 06:51:57.138670 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.132848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:42181 (local address 127.20.120.194:33395)
0621 06:51:57.133042 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:57.133049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:57.133360 (+ 311us) server_negotiation.cc:407] Connection header received
0621 06:51:57.133554 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:57.133558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:57.133612 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:57.133709 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:57.134195 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.134957 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.136859 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.137639 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.137805 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.137876 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:57.138311 (+ 435us) server_negotiation.cc:299] Negotiation successful
0621 06:51:57.138432 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 06:51:57.138707 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.132714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:33755 (local address 127.20.120.193:42497)
0621 06:51:57.132894 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:57.132900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:57.133119 (+ 219us) server_negotiation.cc:407] Connection header received
0621 06:51:57.133350 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:57.133354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:57.133413 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:57.133505 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:51:57.133897 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.134661 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.136840 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.137639 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.137836 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.137944 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:57.138312 (+ 368us) server_negotiation.cc:299] Negotiation successful
0621 06:51:57.138429 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:51:57.138849 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.132568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:33395 (local address 127.20.120.195:42181)
0621 06:51:57.133256 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:57.133270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:57.133405 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:57.133730 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:57.133734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:57.133754 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:57.134056 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:57.134064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.135094 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.135101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:57.136718 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:57.136729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.137809 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.137815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:57.138058 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.138166 (+ 108us) client_negotiation.cc:769] Sending connection context
0621 06:51:57.138275 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 06:51:57.138401 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":59,"thread_start_us":480,"threads_started":1}
I20260621 06:51:57.139022 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.132477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:42497 (local address 127.20.120.195:33755)
0621 06:51:57.133007 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:57.133023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:57.133203 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:57.133534 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:57.133538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:57.133562 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:51:57.133750 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:57.133758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.134800 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.134805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:57.136698 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:57.136709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.137851 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.137857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:57.138162 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.138185 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 06:51:57.138287 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 06:51:57.138414 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":59,"thread_start_us":309,"threads_started":1}
I20260621 06:51:57.139779 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c252660ba145d3ba66c152c1175926" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" is_pre_election: true
I20260621 06:51:57.140028 22875 raft_consensus.cc:2454] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 0.
I20260621 06:51:57.139881 23009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c252660ba145d3ba66c152c1175926" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13057e4f6ce347738b11c51a48d7a8a6" is_pre_election: true
I20260621 06:51:57.140345 23009 raft_consensus.cc:2454] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 0.
I20260621 06:51:57.140515 23080 leader_election.cc:302] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c7c0e9db174615b17d3ecdf41edd2a, ed50517b59b94c54a902a4c4f4252d08; no voters:
I20260621 06:51:57.140934 23224 raft_consensus.cc:2781] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:51:57.141059 23224 raft_consensus.cc:482] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:51:57.141156 23224 raft_consensus.cc:3037] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:57.143417 23224 raft_consensus.cc:504] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:57.143906 23224 leader_election.cc:288] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 election: Requested vote from peers 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497), 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395)
I20260621 06:51:57.144310 23009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c252660ba145d3ba66c152c1175926" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13057e4f6ce347738b11c51a48d7a8a6"
I20260621 06:51:57.144297 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69c252660ba145d3ba66c152c1175926" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c7c0e9db174615b17d3ecdf41edd2a"
I20260621 06:51:57.144484 23009 raft_consensus.cc:3037] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:57.144502 22875 raft_consensus.cc:3037] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:57.147387 22875 raft_consensus.cc:2454] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 1.
I20260621 06:51:57.147403 23009 raft_consensus.cc:2454] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 1.
I20260621 06:51:57.147755 23080 leader_election.cc:302] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c7c0e9db174615b17d3ecdf41edd2a, ed50517b59b94c54a902a4c4f4252d08; no voters:
I20260621 06:51:57.148025 23224 raft_consensus.cc:2781] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:51:57.148372 23224 raft_consensus.cc:686] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [term 1 LEADER]: Becoming Leader. State: Replica: ed50517b59b94c54a902a4c4f4252d08, State: Running, Role: LEADER
I20260621 06:51:57.148795 23224 consensus_queue.cc:237] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:57.152102 23241 tablet_replica.cc:439] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: TxnStatusTablet state changed. Reason: New leader ed50517b59b94c54a902a4c4f4252d08. Latest consensus state: current_term: 1 leader_uuid: "ed50517b59b94c54a902a4c4f4252d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } }
I20260621 06:51:57.152519 23241 tablet_replica.cc:442] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: This TxnStatusTablet replica's current role is: LEADER
I20260621 06:51:57.153138 23243 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 06:51:57.153396 23243 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 06:51:57.153632 22711 catalog_manager.cc:5495] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed50517b59b94c54a902a4c4f4252d08 (127.20.120.195). New cstate: current_term: 1 leader_uuid: "ed50517b59b94c54a902a4c4f4252d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } health_report { overall_health: HEALTHY } } }
I20260621 06:51:57.171885 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.165840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:34799 (local address 127.0.0.1:43422)
0621 06:51:57.166235 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:57.166254 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:57.166402 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:57.166906 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:57.166910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:57.166918 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 06:51:57.167055 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:57.167061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.168448 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.168451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:57.169394 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:57.169404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.170353 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.170357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:57.170475 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.170519 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 06:51:57.171414 (+ 895us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 06:51:57.171427 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 06:51:57.171479 (+ 52us) client_negotiation.cc:240] Negotiation successful
0621 06:51:57.171578 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260621 06:51:57.171936 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.165843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.20.120.195:34799)
0621 06:51:57.166017 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:57.166020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:57.166331 (+ 311us) server_negotiation.cc:407] Connection header received
0621 06:51:57.166631 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:57.166636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:57.166738 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:57.166875 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:51:57.167179 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.168334 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.169523 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.170215 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.170335 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.170704 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:51:57.171321 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:51:57.171450 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:57.171500 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 06:51:57.171572 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 06:51:57.178622 23009 raft_consensus.cc:1261] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Refusing update from remote peer ed50517b59b94c54a902a4c4f4252d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:57.178642 22875 raft_consensus.cc:1261] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Refusing update from remote peer ed50517b59b94c54a902a4c4f4252d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:57.179486 23241 consensus_queue.cc:1035] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:57.179775 23224 consensus_queue.cc:1035] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:57.195833 23223 tablet_replica.cc:439] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6: TxnStatusTablet state changed. Reason: New leader ed50517b59b94c54a902a4c4f4252d08. Latest consensus state: current_term: 1 leader_uuid: "ed50517b59b94c54a902a4c4f4252d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } }
I20260621 06:51:57.196204 23223 tablet_replica.cc:442] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:57.197528 23225 tablet_replica.cc:439] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a: TxnStatusTablet state changed. Reason: New leader ed50517b59b94c54a902a4c4f4252d08. Latest consensus state: current_term: 1 leader_uuid: "ed50517b59b94c54a902a4c4f4252d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } }
I20260621 06:51:57.197803 23225 tablet_replica.cc:442] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:57.211838 23224 tablet_replica.cc:439] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed50517b59b94c54a902a4c4f4252d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } }
I20260621 06:51:57.212105 23224 tablet_replica.cc:442] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: This TxnStatusTablet replica's current role is: LEADER
I20260621 06:51:57.211920 23241 tablet_replica.cc:439] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed50517b59b94c54a902a4c4f4252d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } }
I20260621 06:51:57.212260 23241 tablet_replica.cc:442] T 69c252660ba145d3ba66c152c1175926 P ed50517b59b94c54a902a4c4f4252d08: This TxnStatusTablet replica's current role is: LEADER
I20260621 06:51:57.219118 23223 tablet_replica.cc:439] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed50517b59b94c54a902a4c4f4252d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } }
I20260621 06:51:57.219388 23223 tablet_replica.cc:442] T 69c252660ba145d3ba66c152c1175926 P 13057e4f6ce347738b11c51a48d7a8a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:57.221100 23225 tablet_replica.cc:439] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed50517b59b94c54a902a4c4f4252d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } } }
I20260621 06:51:57.221321 23225 tablet_replica.cc:442] T 69c252660ba145d3ba66c152c1175926 P 87c7c0e9db174615b17d3ecdf41edd2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:51:57.233341 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.226929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35888 (local address 127.20.120.194:33395)
0621 06:51:57.227126 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:57.227136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:57.227635 (+ 499us) server_negotiation.cc:407] Connection header received
0621 06:51:57.227879 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:57.227883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:57.227955 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:57.228065 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:51:57.228528 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.229311 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.231308 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.231704 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.231801 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.232290 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:51:57.232758 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:51:57.232916 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:57.233005 (+ 89us) server_negotiation.cc:299] Negotiation successful
0621 06:51:57.233077 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:51:57.248519 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.243262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:37885 (local address 127.20.120.195:34799)
0621 06:51:57.243438 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:57.243445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:57.243857 (+ 412us) server_negotiation.cc:407] Connection header received
0621 06:51:57.244047 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:57.244056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:57.244125 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:57.244239 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:51:57.244652 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.245357 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.246802 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.247265 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.247420 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.247819 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:51:57.247912 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:51:57.248024 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:57.248159 (+ 135us) server_negotiation.cc:299] Negotiation successful
0621 06:51:57.248230 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 06:51:57.248627 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.243226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:34799 (local address 127.20.120.194:37885)
0621 06:51:57.243767 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:57.243781 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:57.243925 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:57.244243 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:57.244247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:57.244258 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 06:51:57.244513 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:57.244521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.245494 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.245498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:57.246664 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:57.246675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.247447 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.247453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:57.247586 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.247631 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 06:51:57.248025 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 06:51:57.248037 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 06:51:57.248130 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 06:51:57.248247 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":304,"threads_started":1}
I20260621 06:51:57.257748 22875 raft_consensus.cc:1261] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Refusing update from remote peer 13057e4f6ce347738b11c51a48d7a8a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:57.258227 23148 raft_consensus.cc:1261] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Refusing update from remote peer 13057e4f6ce347738b11c51a48d7a8a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:57.258355 23228 consensus_queue.cc:1035] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:57.258927 23223 consensus_queue.cc:1035] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:57.263562 23248 mvcc.cc:205] Tried to move back new op lower bound from 7299173241882120192 to 7299173240246521856. Current Snapshot: MvccSnapshot[applied={T|T < 7299173241882120192}]
I20260621 06:51:57.266834 23246 mvcc.cc:205] Tried to move back new op lower bound from 7299173241882120192 to 7299173240246521856. Current Snapshot: MvccSnapshot[applied={T|T < 7299173241882120192}]
I20260621 06:51:57.291702 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.285446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:36375 (local address 127.20.120.194:33395)
0621 06:51:57.285632 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:57.285636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:57.286350 (+ 714us) server_negotiation.cc:407] Connection header received
0621 06:51:57.286596 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:57.286601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:57.286682 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:57.286756 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:51:57.287342 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.288222 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.289756 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.290368 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.290509 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.291018 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:51:57.291143 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:51:57.291226 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:57.291342 (+ 116us) server_negotiation.cc:299] Negotiation successful
0621 06:51:57.291410 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 06:51:57.292032 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.285521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:33395 (local address 127.20.120.195:36375)
0621 06:51:57.286230 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:57.286248 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:57.286445 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:57.286769 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:57.286773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:57.286784 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 06:51:57.287148 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:57.287179 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.288361 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.288364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:57.289606 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:57.289616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.290549 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.290559 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:57.290733 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.290814 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 06:51:57.291222 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 06:51:57.291232 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 06:51:57.291325 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 06:51:57.291565 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":344,"threads_started":1}
I20260621 06:51:57.329617 22711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:51:57.329880 22711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:51:57.333694 22711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f: Sending DeleteTablet for 3 replicas of tablet 6cf5021577104b609fa7df218a2e872a
I20260621 06:51:57.334730 23128 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf5021577104b609fa7df218a2e872a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:57 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:43414
I20260621 06:51:57.334893 22989 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf5021577104b609fa7df218a2e872a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:57 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:35874
I20260621 06:51:57.335824 22850 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf5021577104b609fa7df218a2e872a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:57 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:46264
I20260621 06:51:57.341785 22712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:51:57.342420 22712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:51:57.349560 22849 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca95f398ef2413498bfedba59667832 (DEFAULT_TABLE table=test-table [id=c2ab6af7bb204c89900547bfe2f50c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:57.350466 23127 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca95f398ef2413498bfedba59667832 (DEFAULT_TABLE table=test-table [id=c2ab6af7bb204c89900547bfe2f50c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:57.351042 23127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca95f398ef2413498bfedba59667832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:57.354586 22849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca95f398ef2413498bfedba59667832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:57.354612 22988 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca95f398ef2413498bfedba59667832 (DEFAULT_TABLE table=test-table [id=c2ab6af7bb204c89900547bfe2f50c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:57.355096 22988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca95f398ef2413498bfedba59667832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:57.357170 23268 tablet_replica.cc:330] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6: stopping tablet replica
I20260621 06:51:57.358655 23268 raft_consensus.cc:2229] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:51:57.359329 23268 raft_consensus.cc:2258] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:57.361311 23268 ts_tablet_manager.cc:1905] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:57.361845 23217 tablet_bootstrap.cc:492] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6: Bootstrap starting.
I20260621 06:51:57.364046 23217 tablet_bootstrap.cc:654] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:57.365444 23218 tablet_bootstrap.cc:492] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a: Bootstrap starting.
I20260621 06:51:57.365584 23219 tablet_bootstrap.cc:492] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08: Bootstrap starting.
I20260621 06:51:57.367097 23218 tablet_bootstrap.cc:654] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:57.367115 23219 tablet_bootstrap.cc:654] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:57.369705 23269 tablet_replica.cc:330] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a: stopping tablet replica
I20260621 06:51:57.370299 23269 raft_consensus.cc:2229] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:51:57.370877 23269 raft_consensus.cc:2258] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:57.371448 23267 tablet_replica.cc:330] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08: stopping tablet replica
I20260621 06:51:57.371922 23267 raft_consensus.cc:2229] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:51:57.372249 23267 raft_consensus.cc:2258] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:57.374426 23217 tablet_bootstrap.cc:492] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6: No bootstrap required, opened a new log
I20260621 06:51:57.374604 23217 ts_tablet_manager.cc:1397] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20260621 06:51:57.375233 23269 ts_tablet_manager.cc:1905] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:57.375334 23217 raft_consensus.cc:348] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:57.375557 23217 raft_consensus.cc:374] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:57.375654 23217 raft_consensus.cc:729] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13057e4f6ce347738b11c51a48d7a8a6, State: Initialized, Role: FOLLOWER
I20260621 06:51:57.375917 23217 consensus_queue.cc:260] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:57.376487 23217 ts_tablet_manager.cc:1428] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 06:51:57.380738 23267 ts_tablet_manager.cc:1905] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:57.386839 23218 tablet_bootstrap.cc:492] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a: No bootstrap required, opened a new log
I20260621 06:51:57.387156 23218 ts_tablet_manager.cc:1397] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.009s
I20260621 06:51:57.388190 23218 raft_consensus.cc:348] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:57.388563 23218 raft_consensus.cc:374] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:57.388713 23218 raft_consensus.cc:729] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c7c0e9db174615b17d3ecdf41edd2a, State: Initialized, Role: FOLLOWER
I20260621 06:51:57.389088 23218 consensus_queue.cc:260] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:57.391039 23218 ts_tablet_manager.cc:1428] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260621 06:51:57.395223 23268 ts_tablet_manager.cc:1918] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:51:57.395649 23269 ts_tablet_manager.cc:1918] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:51:57.395812 23269 log.cc:1196] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/6cf5021577104b609fa7df218a2e872a
I20260621 06:51:57.395807 23268 log.cc:1196] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/6cf5021577104b609fa7df218a2e872a
I20260621 06:51:57.396308 23269 ts_tablet_manager.cc:1939] T 6cf5021577104b609fa7df218a2e872a P 87c7c0e9db174615b17d3ecdf41edd2a: Deleting consensus metadata
I20260621 06:51:57.396512 23268 ts_tablet_manager.cc:1939] T 6cf5021577104b609fa7df218a2e872a P 13057e4f6ce347738b11c51a48d7a8a6: Deleting consensus metadata
I20260621 06:51:57.398077 22698 catalog_manager.cc:4841] TS 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497): tablet 6cf5021577104b609fa7df218a2e872a (table test-table [id=fa16d7304da94e74af27c51dfb0731e3]) successfully deleted
I20260621 06:51:57.398162 22699 catalog_manager.cc:4841] TS 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395): tablet 6cf5021577104b609fa7df218a2e872a (table test-table [id=fa16d7304da94e74af27c51dfb0731e3]) successfully deleted
I20260621 06:51:57.400012 23267 ts_tablet_manager.cc:1918] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:51:57.400177 23267 log.cc:1196] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/6cf5021577104b609fa7df218a2e872a
I20260621 06:51:57.400652 23267 ts_tablet_manager.cc:1939] T 6cf5021577104b609fa7df218a2e872a P ed50517b59b94c54a902a4c4f4252d08: Deleting consensus metadata
I20260621 06:51:57.402159 22697 catalog_manager.cc:4841] TS ed50517b59b94c54a902a4c4f4252d08 (127.20.120.195:34799): tablet 6cf5021577104b609fa7df218a2e872a (table test-table [id=fa16d7304da94e74af27c51dfb0731e3]) successfully deleted
I20260621 06:51:57.403218 23219 tablet_bootstrap.cc:492] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08: No bootstrap required, opened a new log
I20260621 06:51:57.403395 23219 ts_tablet_manager.cc:1397] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.008s
I20260621 06:51:57.404165 23219 raft_consensus.cc:348] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:57.404407 23219 raft_consensus.cc:374] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:57.404487 23219 raft_consensus.cc:729] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed50517b59b94c54a902a4c4f4252d08, State: Initialized, Role: FOLLOWER
I20260621 06:51:57.404736 23219 consensus_queue.cc:260] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:57.405335 23219 ts_tablet_manager.cc:1428] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 06:51:57.511233 23241 raft_consensus.cc:482] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:51:57.511482 23241 raft_consensus.cc:504] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:57.512305 23241 leader_election.cc:288] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497), 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395)
I20260621 06:51:57.512897 23009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca95f398ef2413498bfedba59667832" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13057e4f6ce347738b11c51a48d7a8a6" is_pre_election: true
I20260621 06:51:57.513144 23009 raft_consensus.cc:2454] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 0.
I20260621 06:51:57.513741 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca95f398ef2413498bfedba59667832" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" is_pre_election: true
I20260621 06:51:57.513983 22875 raft_consensus.cc:2454] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 0.
I20260621 06:51:57.514474 23081 leader_election.cc:302] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13057e4f6ce347738b11c51a48d7a8a6, ed50517b59b94c54a902a4c4f4252d08; no voters:
I20260621 06:51:57.515292 23241 raft_consensus.cc:2781] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:51:57.515388 23241 raft_consensus.cc:482] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:51:57.515486 23241 raft_consensus.cc:3037] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:57.518647 23241 raft_consensus.cc:504] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:57.519819 22875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca95f398ef2413498bfedba59667832" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c7c0e9db174615b17d3ecdf41edd2a"
I20260621 06:51:57.520001 22875 raft_consensus.cc:3037] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:57.523093 22875 raft_consensus.cc:2454] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 1.
I20260621 06:51:57.523600 23080 leader_election.cc:302] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c7c0e9db174615b17d3ecdf41edd2a, ed50517b59b94c54a902a4c4f4252d08; no voters:
I20260621 06:51:57.523877 23251 raft_consensus.cc:2781] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:51:57.523998 23251 raft_consensus.cc:686] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 1 LEADER]: Becoming Leader. State: Replica: ed50517b59b94c54a902a4c4f4252d08, State: Running, Role: LEADER
I20260621 06:51:57.524351 23251 consensus_queue.cc:237] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:57.527684 23009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca95f398ef2413498bfedba59667832" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13057e4f6ce347738b11c51a48d7a8a6"
I20260621 06:51:57.527904 23009 raft_consensus.cc:3037] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:57.531087 23009 raft_consensus.cc:2454] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 1.
I20260621 06:51:57.531917 23241 leader_election.cc:288] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 election: Requested vote from peers 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497), 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395)
I20260621 06:51:57.540290 22711 catalog_manager.cc:5495] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed50517b59b94c54a902a4c4f4252d08 (127.20.120.195). New cstate: current_term: 1 leader_uuid: "ed50517b59b94c54a902a4c4f4252d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } health_report { overall_health: HEALTHY } } }
I20260621 06:51:57.611176 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.604849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.20.120.195:34799)
0621 06:51:57.605029 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:57.605035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:57.605325 (+ 290us) server_negotiation.cc:407] Connection header received
0621 06:51:57.605530 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:57.605535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:57.605632 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:57.605715 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:51:57.606171 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.607250 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.609102 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.609764 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.609877 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.610364 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:51:57.610511 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:51:57.610608 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:57.610760 (+ 152us) server_negotiation.cc:299] Negotiation successful
0621 06:51:57.610846 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 06:51:57.627386 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.620814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:33313 (local address 127.20.120.195:34799)
0621 06:51:57.620970 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:57.620977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:57.621183 (+ 206us) server_negotiation.cc:407] Connection header received
0621 06:51:57.621325 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:57.621330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:57.621409 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:57.621493 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:51:57.621900 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.623018 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.625027 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.625700 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.625844 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.626450 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:51:57.626590 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:51:57.626704 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:57.626891 (+ 187us) server_negotiation.cc:299] Negotiation successful
0621 06:51:57.626996 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":40}
I20260621 06:51:57.627388 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:57.620885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:34799 (local address 127.20.120.195:33313)
0621 06:51:57.621094 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 06:51:57.621111 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:51:57.621212 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:51:57.621529 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:51:57.621551 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:51:57.621564 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 06:51:57.621772 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:57.621781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.623176 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.623182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:57.624883 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:51:57.624907 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:51:57.625855 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:51:57.625861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:51:57.626072 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:57.626118 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 06:51:57.626728 (+ 610us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 06:51:57.626745 (+ 17us) client_negotiation.cc:769] Sending connection context
0621 06:51:57.626839 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 06:51:57.626996 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":27}
I20260621 06:51:57.637562 23009 raft_consensus.cc:1261] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Refusing update from remote peer ed50517b59b94c54a902a4c4f4252d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:57.637616 22875 raft_consensus.cc:1261] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Refusing update from remote peer ed50517b59b94c54a902a4c4f4252d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:57.638177 23241 consensus_queue.cc:1035] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:57.638506 23224 consensus_queue.cc:1035] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:57.655650 23247 mvcc.cc:205] Tried to move back new op lower bound from 7299173243439169536 to 7299173242983022592. Current Snapshot: MvccSnapshot[applied={T|T < 7299173243439169536}]
W20260621 06:51:57.669816 23246 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 06:51:59.281044 22711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:51:59.281339 22711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:51:59.285027 22711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f: Sending DeleteTablet for 3 replicas of tablet 8ca95f398ef2413498bfedba59667832
I20260621 06:51:59.285777 22850 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca95f398ef2413498bfedba59667832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:59 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:46264
I20260621 06:51:59.285856 22988 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca95f398ef2413498bfedba59667832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:59 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:35874
I20260621 06:51:59.285871 23128 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca95f398ef2413498bfedba59667832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:59 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:43414
I20260621 06:51:59.298521 23297 tablet_replica.cc:330] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6: stopping tablet replica
I20260621 06:51:59.299134 23298 tablet_replica.cc:330] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a: stopping tablet replica
I20260621 06:51:59.299479 23298 raft_consensus.cc:2229] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:51:59.299774 23297 raft_consensus.cc:2229] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:51:59.299844 23296 tablet_replica.cc:330] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08: stopping tablet replica
I20260621 06:51:59.299846 23298 raft_consensus.cc:2258] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:59.300086 23297 raft_consensus.cc:2258] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:59.300064 23296 raft_consensus.cc:2229] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:51:59.301108 23297 ts_tablet_manager.cc:1905] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:59.301097 23296 raft_consensus.cc:2258] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:59.301271 23298 ts_tablet_manager.cc:1905] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:59.302220 23296 ts_tablet_manager.cc:1905] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:59.308249 23297 ts_tablet_manager.cc:1918] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 06:51:59.308431 23297 log.cc:1196] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/8ca95f398ef2413498bfedba59667832
I20260621 06:51:59.308916 23297 ts_tablet_manager.cc:1939] T 8ca95f398ef2413498bfedba59667832 P 13057e4f6ce347738b11c51a48d7a8a6: Deleting consensus metadata
I20260621 06:51:59.308988 23298 ts_tablet_manager.cc:1918] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 06:51:59.309137 23298 log.cc:1196] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/8ca95f398ef2413498bfedba59667832
I20260621 06:51:59.309613 23298 ts_tablet_manager.cc:1939] T 8ca95f398ef2413498bfedba59667832 P 87c7c0e9db174615b17d3ecdf41edd2a: Deleting consensus metadata
I20260621 06:51:59.310441 23296 ts_tablet_manager.cc:1918] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 06:51:59.310652 23296 log.cc:1196] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/8ca95f398ef2413498bfedba59667832
I20260621 06:51:59.310701 22699 catalog_manager.cc:4841] TS 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395): tablet 8ca95f398ef2413498bfedba59667832 (table test-table [id=c2ab6af7bb204c89900547bfe2f50c8c]) successfully deleted
I20260621 06:51:59.310701 22698 catalog_manager.cc:4841] TS 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497): tablet 8ca95f398ef2413498bfedba59667832 (table test-table [id=c2ab6af7bb204c89900547bfe2f50c8c]) successfully deleted
I20260621 06:51:59.311789 23296 ts_tablet_manager.cc:1939] T 8ca95f398ef2413498bfedba59667832 P ed50517b59b94c54a902a4c4f4252d08: Deleting consensus metadata
I20260621 06:51:59.312995 22697 catalog_manager.cc:4841] TS ed50517b59b94c54a902a4c4f4252d08 (127.20.120.195:34799): tablet 8ca95f398ef2413498bfedba59667832 (table test-table [id=c2ab6af7bb204c89900547bfe2f50c8c]) successfully deleted
Jun 21 06:51:59 dist-test-slave-gmf2 krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 06:51:59 dist-test-slave-gmf2 krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:51:59.346876 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:59.330599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.20.120.254:44355)
0621 06:51:59.331168 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:59.331173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:59.331231 (+ 58us) server_negotiation.cc:407] Connection header received
0621 06:51:59.331382 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:59.331386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:59.331447 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:59.331536 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:51:59.332768 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:59.333515 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:59.334701 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:59.335067 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:59.335165 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:59.342318 (+ 7153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:51:59.342323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:51:59.342326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:51:59.342358 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:51:59.344690 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:59.345281 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:59.345284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:59.345286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:59.345393 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:51:59.345671 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:51:59.345674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:51:59.345676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:51:59.345933 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:51:59.346111 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:59.346342 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 06:51:59.346489 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":294,"threads_started":1}
I20260621 06:51:59.350060 22711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:51:59.350569 22711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:51:59.356709 22850 tablet_service.cc:1467] Processing CreateTablet for tablet 5fac13dd063b466ead3a834e083dd19b (DEFAULT_TABLE table=test-table [id=317173061a714a5f902a0112a04a8a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:59.357012 22988 tablet_service.cc:1467] Processing CreateTablet for tablet 5fac13dd063b466ead3a834e083dd19b (DEFAULT_TABLE table=test-table [id=317173061a714a5f902a0112a04a8a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:59.357157 22850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fac13dd063b466ead3a834e083dd19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:59.357077 23128 tablet_service.cc:1467] Processing CreateTablet for tablet 5fac13dd063b466ead3a834e083dd19b (DEFAULT_TABLE table=test-table [id=317173061a714a5f902a0112a04a8a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:59.357424 22988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fac13dd063b466ead3a834e083dd19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:59.357434 23128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fac13dd063b466ead3a834e083dd19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:59.362830 23307 tablet_bootstrap.cc:492] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a: Bootstrap starting.
I20260621 06:51:59.364557 23307 tablet_bootstrap.cc:654] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:59.364979 23308 tablet_bootstrap.cc:492] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6: Bootstrap starting.
I20260621 06:51:59.365245 23309 tablet_bootstrap.cc:492] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08: Bootstrap starting.
I20260621 06:51:59.366655 23308 tablet_bootstrap.cc:654] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:59.366958 23307 tablet_bootstrap.cc:492] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a: No bootstrap required, opened a new log
I20260621 06:51:59.367126 23307 ts_tablet_manager.cc:1397] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 06:51:59.367573 23309 tablet_bootstrap.cc:654] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:59.368517 23307 raft_consensus.cc:348] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.368971 23307 raft_consensus.cc:374] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:59.369112 23307 raft_consensus.cc:729] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c7c0e9db174615b17d3ecdf41edd2a, State: Initialized, Role: FOLLOWER
I20260621 06:51:59.369388 23307 consensus_queue.cc:260] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.370092 23308 tablet_bootstrap.cc:492] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6: No bootstrap required, opened a new log
I20260621 06:51:59.370107 23309 tablet_bootstrap.cc:492] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08: No bootstrap required, opened a new log
I20260621 06:51:59.370234 23308 ts_tablet_manager.cc:1397] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260621 06:51:59.370249 23309 ts_tablet_manager.cc:1397] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 06:51:59.370487 23307 ts_tablet_manager.cc:1428] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 06:51:59.371091 23309 raft_consensus.cc:348] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.371160 23308 raft_consensus.cc:348] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.371392 23308 raft_consensus.cc:374] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:59.371389 23309 raft_consensus.cc:374] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:59.371485 23308 raft_consensus.cc:729] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13057e4f6ce347738b11c51a48d7a8a6, State: Initialized, Role: FOLLOWER
I20260621 06:51:59.371490 23309 raft_consensus.cc:729] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed50517b59b94c54a902a4c4f4252d08, State: Initialized, Role: FOLLOWER
I20260621 06:51:59.371734 23309 consensus_queue.cc:260] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.371752 23308 consensus_queue.cc:260] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.372830 23309 ts_tablet_manager.cc:1428] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 06:51:59.374514 23308 ts_tablet_manager.cc:1428] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260621 06:51:59.389654 23279 raft_consensus.cc:482] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:51:59.389802 23279 raft_consensus.cc:504] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.390515 23279 leader_election.cc:288] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497), 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395)
I20260621 06:51:59.390918 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fac13dd063b466ead3a834e083dd19b" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" is_pre_election: true
I20260621 06:51:59.391042 23008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fac13dd063b466ead3a834e083dd19b" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13057e4f6ce347738b11c51a48d7a8a6" is_pre_election: true
I20260621 06:51:59.391156 22874 raft_consensus.cc:2454] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 0.
I20260621 06:51:59.391273 23008 raft_consensus.cc:2454] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 0.
I20260621 06:51:59.391590 23080 leader_election.cc:302] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c7c0e9db174615b17d3ecdf41edd2a, ed50517b59b94c54a902a4c4f4252d08; no voters:
I20260621 06:51:59.391848 23279 raft_consensus.cc:2781] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:51:59.391973 23279 raft_consensus.cc:482] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:51:59.392438 23279 raft_consensus.cc:3037] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:59.394636 23279 raft_consensus.cc:504] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.395152 23279 leader_election.cc:288] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 election: Requested vote from peers 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497), 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395)
I20260621 06:51:59.395507 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fac13dd063b466ead3a834e083dd19b" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c7c0e9db174615b17d3ecdf41edd2a"
I20260621 06:51:59.395613 23008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fac13dd063b466ead3a834e083dd19b" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13057e4f6ce347738b11c51a48d7a8a6"
I20260621 06:51:59.395726 22874 raft_consensus.cc:3037] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:59.395774 23008 raft_consensus.cc:3037] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:59.398581 23008 raft_consensus.cc:2454] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 1.
I20260621 06:51:59.398587 22874 raft_consensus.cc:2454] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 1.
I20260621 06:51:59.398952 23081 leader_election.cc:302] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13057e4f6ce347738b11c51a48d7a8a6, ed50517b59b94c54a902a4c4f4252d08; no voters:
I20260621 06:51:59.399235 23279 raft_consensus.cc:2781] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:51:59.399394 23279 raft_consensus.cc:686] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 1 LEADER]: Becoming Leader. State: Replica: ed50517b59b94c54a902a4c4f4252d08, State: Running, Role: LEADER
I20260621 06:51:59.399662 23279 consensus_queue.cc:237] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.401934 22711 catalog_manager.cc:5495] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed50517b59b94c54a902a4c4f4252d08 (127.20.120.195). New cstate: current_term: 1 leader_uuid: "ed50517b59b94c54a902a4c4f4252d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } health_report { overall_health: HEALTHY } } }
I20260621 06:51:59.429809 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:51:59.424733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.20.120.195:34799)
0621 06:51:59.425319 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0621 06:51:59.425323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:51:59.425341 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:51:59.425386 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:51:59.425389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:51:59.425441 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:51:59.425519 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:51:59.425864 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:59.426613 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:59.428061 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:51:59.428472 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:51:59.428565 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:51:59.429058 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:51:59.429164 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:51:59.429262 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 06:51:59.429424 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 06:51:59.429495 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":371,"threads_started":1}
I20260621 06:51:59.439043 23008 raft_consensus.cc:1261] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Refusing update from remote peer ed50517b59b94c54a902a4c4f4252d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:59.439226 22874 raft_consensus.cc:1261] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Refusing update from remote peer ed50517b59b94c54a902a4c4f4252d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:59.439679 23279 consensus_queue.cc:1035] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:59.440091 23284 consensus_queue.cc:1035] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:59.488929 22711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:51:59.489159 22711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:51:59.491858 22711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f: Sending DeleteTablet for 3 replicas of tablet 5fac13dd063b466ead3a834e083dd19b
I20260621 06:51:59.492564 22850 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fac13dd063b466ead3a834e083dd19b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:59 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:46264
I20260621 06:51:59.492762 23127 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fac13dd063b466ead3a834e083dd19b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:59 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:43414
I20260621 06:51:59.492772 22988 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fac13dd063b466ead3a834e083dd19b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:51:59 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:35874
I20260621 06:51:59.493036 23298 tablet_replica.cc:330] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a: stopping tablet replica
I20260621 06:51:59.493038 23296 tablet_replica.cc:330] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08: stopping tablet replica
I20260621 06:51:59.493068 23297 tablet_replica.cc:330] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6: stopping tablet replica
I20260621 06:51:59.493266 23298 raft_consensus.cc:2229] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:51:59.493273 23296 raft_consensus.cc:2229] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:51:59.493273 23297 raft_consensus.cc:2229] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:51:59.493440 23297 raft_consensus.cc:2258] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:59.493443 23298 raft_consensus.cc:2258] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:59.493556 23296 raft_consensus.cc:2258] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:51:59.494590 23298 ts_tablet_manager.cc:1905] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:59.494757 23296 ts_tablet_manager.cc:1905] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:59.494875 23297 ts_tablet_manager.cc:1905] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:51:59.494810 22711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:51:59.495404 22711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:51:59.501266 23297 ts_tablet_manager.cc:1918] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:51:59.501400 23297 log.cc:1196] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/5fac13dd063b466ead3a834e083dd19b
I20260621 06:51:59.501753 23297 ts_tablet_manager.cc:1939] T 5fac13dd063b466ead3a834e083dd19b P 13057e4f6ce347738b11c51a48d7a8a6: Deleting consensus metadata
I20260621 06:51:59.502195 23296 ts_tablet_manager.cc:1918] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:51:59.502336 23296 log.cc:1196] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/5fac13dd063b466ead3a834e083dd19b
I20260621 06:51:59.502796 23296 ts_tablet_manager.cc:1939] T 5fac13dd063b466ead3a834e083dd19b P ed50517b59b94c54a902a4c4f4252d08: Deleting consensus metadata
I20260621 06:51:59.502816 22699 catalog_manager.cc:4841] TS 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395): tablet 5fac13dd063b466ead3a834e083dd19b (table test-table [id=317173061a714a5f902a0112a04a8a87]) successfully deleted
I20260621 06:51:59.504312 22697 catalog_manager.cc:4841] TS ed50517b59b94c54a902a4c4f4252d08 (127.20.120.195:34799): tablet 5fac13dd063b466ead3a834e083dd19b (table test-table [id=317173061a714a5f902a0112a04a8a87]) successfully deleted
I20260621 06:51:59.504357 22850 tablet_service.cc:1467] Processing CreateTablet for tablet 5a59a777f29843b089cd52244ee3c8bb (DEFAULT_TABLE table=test-table [id=60a3ffdb221a46519b244aff8319403c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:59.504438 22988 tablet_service.cc:1467] Processing CreateTablet for tablet 5a59a777f29843b089cd52244ee3c8bb (DEFAULT_TABLE table=test-table [id=60a3ffdb221a46519b244aff8319403c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:59.504799 22850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a59a777f29843b089cd52244ee3c8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:59.504806 22988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a59a777f29843b089cd52244ee3c8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:59.504850 23298 ts_tablet_manager.cc:1918] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:51:59.505012 23298 log.cc:1196] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/5fac13dd063b466ead3a834e083dd19b
I20260621 06:51:59.505043 23127 tablet_service.cc:1467] Processing CreateTablet for tablet 5a59a777f29843b089cd52244ee3c8bb (DEFAULT_TABLE table=test-table [id=60a3ffdb221a46519b244aff8319403c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:51:59.505436 23298 ts_tablet_manager.cc:1939] T 5fac13dd063b466ead3a834e083dd19b P 87c7c0e9db174615b17d3ecdf41edd2a: Deleting consensus metadata
I20260621 06:51:59.505446 23127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a59a777f29843b089cd52244ee3c8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:51:59.506395 22698 catalog_manager.cc:4841] TS 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497): tablet 5fac13dd063b466ead3a834e083dd19b (table test-table [id=317173061a714a5f902a0112a04a8a87]) successfully deleted
I20260621 06:51:59.509490 23308 tablet_bootstrap.cc:492] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6: Bootstrap starting.
I20260621 06:51:59.510819 23307 tablet_bootstrap.cc:492] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a: Bootstrap starting.
I20260621 06:51:59.511059 23308 tablet_bootstrap.cc:654] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:59.512216 23309 tablet_bootstrap.cc:492] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08: Bootstrap starting.
I20260621 06:51:59.512872 23307 tablet_bootstrap.cc:654] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:59.513659 23309 tablet_bootstrap.cc:654] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08: Neither blocks nor log segments found. Creating new log.
I20260621 06:51:59.515060 23307 tablet_bootstrap.cc:492] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a: No bootstrap required, opened a new log
I20260621 06:51:59.515225 23307 ts_tablet_manager.cc:1397] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 06:51:59.515913 23307 raft_consensus.cc:348] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.516160 23307 raft_consensus.cc:374] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:59.516235 23307 raft_consensus.cc:729] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c7c0e9db174615b17d3ecdf41edd2a, State: Initialized, Role: FOLLOWER
I20260621 06:51:59.516446 23307 consensus_queue.cc:260] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.516963 23307 ts_tablet_manager.cc:1428] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 06:51:59.522562 23308 tablet_bootstrap.cc:492] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6: No bootstrap required, opened a new log
I20260621 06:51:59.522748 23308 ts_tablet_manager.cc:1397] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260621 06:51:59.523568 23308 raft_consensus.cc:348] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.523842 23308 raft_consensus.cc:374] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:59.523917 23308 raft_consensus.cc:729] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13057e4f6ce347738b11c51a48d7a8a6, State: Initialized, Role: FOLLOWER
I20260621 06:51:59.524163 23308 consensus_queue.cc:260] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.524684 23308 ts_tablet_manager.cc:1428] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 06:51:59.535254 23309 tablet_bootstrap.cc:492] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08: No bootstrap required, opened a new log
I20260621 06:51:59.535427 23309 ts_tablet_manager.cc:1397] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.020s
I20260621 06:51:59.536182 23309 raft_consensus.cc:348] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.536440 23309 raft_consensus.cc:374] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:51:59.536510 23309 raft_consensus.cc:729] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed50517b59b94c54a902a4c4f4252d08, State: Initialized, Role: FOLLOWER
I20260621 06:51:59.536774 23309 consensus_queue.cc:260] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.537391 23309 ts_tablet_manager.cc:1428] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 06:51:59.727528 23284 raft_consensus.cc:482] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:51:59.727788 23284 raft_consensus.cc:504] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.728406 23284 leader_election.cc:288] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497), 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395)
I20260621 06:51:59.728991 23009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a59a777f29843b089cd52244ee3c8bb" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13057e4f6ce347738b11c51a48d7a8a6" is_pre_election: true
I20260621 06:51:59.729086 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a59a777f29843b089cd52244ee3c8bb" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" is_pre_election: true
I20260621 06:51:59.729245 23009 raft_consensus.cc:2454] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 0.
I20260621 06:51:59.729383 22874 raft_consensus.cc:2454] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 0.
I20260621 06:51:59.729702 23081 leader_election.cc:302] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13057e4f6ce347738b11c51a48d7a8a6, ed50517b59b94c54a902a4c4f4252d08; no voters:
I20260621 06:51:59.730014 23284 raft_consensus.cc:2781] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:51:59.730111 23284 raft_consensus.cc:482] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:51:59.730240 23284 raft_consensus.cc:3037] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:59.732204 23284 raft_consensus.cc:504] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.732765 23284 leader_election.cc:288] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 election: Requested vote from peers 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497), 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395)
I20260621 06:51:59.733183 23009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a59a777f29843b089cd52244ee3c8bb" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13057e4f6ce347738b11c51a48d7a8a6"
I20260621 06:51:59.733182 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a59a777f29843b089cd52244ee3c8bb" candidate_uuid: "ed50517b59b94c54a902a4c4f4252d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c7c0e9db174615b17d3ecdf41edd2a"
I20260621 06:51:59.733399 22874 raft_consensus.cc:3037] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:59.733399 23009 raft_consensus.cc:3037] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:51:59.736245 23009 raft_consensus.cc:2454] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 1.
I20260621 06:51:59.736245 22874 raft_consensus.cc:2454] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed50517b59b94c54a902a4c4f4252d08 in term 1.
I20260621 06:51:59.736634 23080 leader_election.cc:302] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c7c0e9db174615b17d3ecdf41edd2a, ed50517b59b94c54a902a4c4f4252d08; no voters:
I20260621 06:51:59.736865 23284 raft_consensus.cc:2781] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:51:59.737028 23284 raft_consensus.cc:686] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 1 LEADER]: Becoming Leader. State: Replica: ed50517b59b94c54a902a4c4f4252d08, State: Running, Role: LEADER
I20260621 06:51:59.737303 23284 consensus_queue.cc:237] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } }
I20260621 06:51:59.739833 22713 catalog_manager.cc:5495] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed50517b59b94c54a902a4c4f4252d08 (127.20.120.195). New cstate: current_term: 1 leader_uuid: "ed50517b59b94c54a902a4c4f4252d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed50517b59b94c54a902a4c4f4252d08" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 34799 } health_report { overall_health: HEALTHY } } }
I20260621 06:51:59.764659 23009 raft_consensus.cc:1261] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Refusing update from remote peer ed50517b59b94c54a902a4c4f4252d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:59.764683 22874 raft_consensus.cc:1261] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Refusing update from remote peer ed50517b59b94c54a902a4c4f4252d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:51:59.765143 23321 consensus_queue.cc:1035] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13057e4f6ce347738b11c51a48d7a8a6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:51:59.765472 23279 consensus_queue.cc:1035] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c7c0e9db174615b17d3ecdf41edd2a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 42497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 06:51:59.783171 23277 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 06:52:01.375402 22711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:01.375636 22711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:01.379586 22711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f55de3a8a62a4c8dbdaeeb87f6eb411f: Sending DeleteTablet for 3 replicas of tablet 5a59a777f29843b089cd52244ee3c8bb
I20260621 06:52:01.380326 22850 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a59a777f29843b089cd52244ee3c8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:01 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:46264
I20260621 06:52:01.380426 23127 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a59a777f29843b089cd52244ee3c8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:01 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:43414
I20260621 06:52:01.380584 22988 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a59a777f29843b089cd52244ee3c8bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:01 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:35874
I20260621 06:52:01.381209 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 22784
I20260621 06:52:01.393752 23348 tablet_replica.cc:330] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08: stopping tablet replica
I20260621 06:52:01.394068 23348 raft_consensus.cc:2229] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:52:01.394515 23348 raft_consensus.cc:2258] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:01.394593 23347 tablet_replica.cc:330] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a: stopping tablet replica
I20260621 06:52:01.394760 23349 tablet_replica.cc:330] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6: stopping tablet replica
I20260621 06:52:01.394774 23347 raft_consensus.cc:2229] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:01.394999 23347 raft_consensus.cc:2258] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:01.395053 23349 raft_consensus.cc:2229] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:01.395399 23349 raft_consensus.cc:2258] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:01.395941 23347 ts_tablet_manager.cc:1905] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:01.395941 23348 ts_tablet_manager.cc:1905] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:01.396445 23349 ts_tablet_manager.cc:1905] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:01.402354 23348 ts_tablet_manager.cc:1918] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 06:52:01.402411 23347 ts_tablet_manager.cc:1918] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 06:52:01.402556 23347 log.cc:1196] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/5a59a777f29843b089cd52244ee3c8bb
I20260621 06:52:01.403023 23347 ts_tablet_manager.cc:1939] T 5a59a777f29843b089cd52244ee3c8bb P 87c7c0e9db174615b17d3ecdf41edd2a: Deleting consensus metadata
I20260621 06:52:01.403018 23348 log.cc:1196] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/5a59a777f29843b089cd52244ee3c8bb
I20260621 06:52:01.403326 23349 ts_tablet_manager.cc:1918] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 06:52:01.403517 23349 log.cc:1196] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/5a59a777f29843b089cd52244ee3c8bb
I20260621 06:52:01.403923 23348 ts_tablet_manager.cc:1939] T 5a59a777f29843b089cd52244ee3c8bb P ed50517b59b94c54a902a4c4f4252d08: Deleting consensus metadata
I20260621 06:52:01.404042 23349 ts_tablet_manager.cc:1939] T 5a59a777f29843b089cd52244ee3c8bb P 13057e4f6ce347738b11c51a48d7a8a6: Deleting consensus metadata
I20260621 06:52:01.404552 22698 catalog_manager.cc:4841] TS 87c7c0e9db174615b17d3ecdf41edd2a (127.20.120.193:42497): tablet 5a59a777f29843b089cd52244ee3c8bb (table test-table [id=60a3ffdb221a46519b244aff8319403c]) successfully deleted
Jun 21 06:52:01 dist-test-slave-gmf2 krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:52:01.404906 22697 catalog_manager.cc:4841] TS ed50517b59b94c54a902a4c4f4252d08 (127.20.120.195:34799): tablet 5a59a777f29843b089cd52244ee3c8bb (table test-table [id=60a3ffdb221a46519b244aff8319403c]) successfully deleted
I20260621 06:52:01.405308 22699 catalog_manager.cc:4841] TS 13057e4f6ce347738b11c51a48d7a8a6 (127.20.120.194:33395): tablet 5a59a777f29843b089cd52244ee3c8bb (table test-table [id=60a3ffdb221a46519b244aff8319403c]) successfully deleted
I20260621 06:52:01.411062 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:01.383027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46268 (local address 127.20.120.193:42497)
0621 06:52:01.393192 (+ 10165us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:01.393202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:01.393786 (+ 584us) server_negotiation.cc:407] Connection header received
0621 06:52:01.394036 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:01.394040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:01.394101 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:01.394228 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:01.396744 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:01.398049 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:01.399759 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:01.400492 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:01.400596 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:01.405965 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:01.405980 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:01.405996 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:01.406026 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:01.408844 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:01.409403 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:01.409410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:01.409412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:01.409541 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:01.409871 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:01.409873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:01.409875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:01.410252 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:01.410473 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:01.410680 (+ 207us) server_negotiation.cc:299] Negotiation successful
0621 06:52:01.410826 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9977,"thread_start_us":9811,"threads_started":1}
W20260621 06:52:01.411787 22915 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46268
W20260621 06:52:01.412371 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 22784: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:52:01.412585 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 22784
I20260621 06:52:01.430236 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 22924
Jun 21 06:52:01 dist-test-slave-gmf2 krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:52:01.446683 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:01.430979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.20.120.194:33395)
0621 06:52:01.431457 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:01.431461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:01.431474 (+ 13us) server_negotiation.cc:407] Connection header received
0621 06:52:01.431522 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:01.431525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:01.431578 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:01.431713 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:01.432895 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:01.433635 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:01.435015 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:01.435764 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:01.435910 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:01.441591 (+ 5681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:01.441634 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:01.441657 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:01.441713 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:01.444301 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:01.444974 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:01.444981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:01.444983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:01.445106 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:01.445439 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:01.445442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:01.445444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:01.445776 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:01.446057 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:01.446284 (+ 227us) server_negotiation.cc:299] Negotiation successful
0621 06:52:01.446440 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":250,"threads_started":1}
W20260621 06:52:01.447322 23054 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35898
W20260621 06:52:01.447911 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 22924: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:52:01.448055 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 22924
I20260621 06:52:01.469604 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 23063
Jun 21 06:52:01 dist-test-slave-gmf2 krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:52:01.484333 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:01.470410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.20.120.195:34799)
0621 06:52:01.470960 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:01.470964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:01.470980 (+ 16us) server_negotiation.cc:407] Connection header received
0621 06:52:01.471039 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:01.471043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:01.471095 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:01.471189 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:01.472483 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:01.473182 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:01.474427 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:01.474822 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:01.474925 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:01.479010 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:01.479025 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:01.479041 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:01.479071 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:01.481853 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:01.482570 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:01.482577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:01.482580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:01.482714 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:01.483106 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:01.483109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:01.483111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:01.483430 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:01.483658 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:01.483923 (+ 265us) server_negotiation.cc:299] Negotiation successful
0621 06:52:01.484081 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":281,"threads_started":1}
W20260621 06:52:01.484951 23193 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43440
W20260621 06:52:01.485534 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 23063: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:52:01.485709 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 23063
I20260621 06:52:01.510073 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 22681
I20260621 06:52:01.511225 22753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:01.623409 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 22681
2026-06-21T06:52:01Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8357 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:52:01 dist-test-slave-gmf2 krb5kdc[23357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:52:01 dist-test-slave-gmf2 krb5kdc[23357](info): set up 2 sockets
Jun 21 06:52:01 dist-test-slave-gmf2 krb5kdc[23357](info): commencing operation
krb5kdc: starting...
W20260621 06:52:02.694141 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:52:02 dist-test-slave-gmf2 krb5kdc[23357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:52:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:52:02.871224 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:45361
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:45361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:03.179680 23370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:03.179944 23370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:03.180066 23370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:03.187103 23370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:03.187209 23370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:03.187309 23370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:03.187400 23370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:03.187479 23370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:03.196962 23370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:45361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:45361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:03.198334 23370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:03.199594 23370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:03.210939 23375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:03.210994 23376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:03.211024 23378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:03.211807 23370 server_base.cc:1034] running on GCE node
I20260621 06:52:03.212538 23370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:03.213819 23370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:03.215042 23370 hybrid_clock.cc:648] HybridClock initialized: now 1782024723215007 us; error 33 us; skew 500 ppm
I20260621 06:52:03.215382 23370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:03 dist-test-slave-gmf2 krb5kdc[23357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024723, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:03.218253 23370 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:52:03.219571 23370 webserver.cc:460] Webserver started at http://127.20.120.254:37557/ using document root <none> and password file <none>
I20260621 06:52:03.220176 23370 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:03.220268 23370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:03.220506 23370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:03.223264 23370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "b201a2367954475c94b63757627835dd"
format_stamp: "Formatted at 2026-06-21 06:52:03 on dist-test-slave-gmf2"
server_key: "80e89460c74226540cf3acfb6abe808f"
server_key_iv: "8b0829c9858335fd51f9f1f1fd1a77ef"
server_key_version: "encryptionkey@0"
I20260621 06:52:03.223901 23370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "b201a2367954475c94b63757627835dd"
format_stamp: "Formatted at 2026-06-21 06:52:03 on dist-test-slave-gmf2"
server_key: "80e89460c74226540cf3acfb6abe808f"
server_key_iv: "8b0829c9858335fd51f9f1f1fd1a77ef"
server_key_version: "encryptionkey@0"
I20260621 06:52:03.228598 23370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 06:52:03.231633 23385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:03.232864 23370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 06:52:03.233045 23370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "b201a2367954475c94b63757627835dd"
format_stamp: "Formatted at 2026-06-21 06:52:03 on dist-test-slave-gmf2"
server_key: "80e89460c74226540cf3acfb6abe808f"
server_key_iv: "8b0829c9858335fd51f9f1f1fd1a77ef"
server_key_version: "encryptionkey@0"
I20260621 06:52:03.233227 23370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:03.248167 23370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:03.251987 23370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:03.252277 23370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:03.270157 23370 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:45361
I20260621 06:52:03.270177 23436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:45361 every 8 connection(s)
I20260621 06:52:03.271479 23370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:03.275091 23437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:03.276975 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 23370
I20260621 06:52:03.277207 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:03.278021 20963 external_mini_cluster.cc:1428] Setting key aac2be4aed680c7e26d986d14094aaa5
I20260621 06:52:03.287384 23437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd: Bootstrap starting.
I20260621 06:52:03.291190 23437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd: Neither blocks nor log segments found. Creating new log.
Jun 21 06:52:03 dist-test-slave-gmf2 krb5kdc[23357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:03.292300 23437 log.cc:824] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:03.295274 23437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd: No bootstrap required, opened a new log
I20260621 06:52:03.300168 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:03.279994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.20.120.254:45361)
0621 06:52:03.280734 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:03.280748 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:03.280786 (+ 38us) server_negotiation.cc:407] Connection header received
0621 06:52:03.282498 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:03.282519 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:03.282877 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:03.283192 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:03.284846 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:03.286193 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:03.287899 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:03.288526 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:03.288647 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:03.293269 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:03.293277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:03.293293 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:03.293327 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:03.296286 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:03.296918 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:03.296924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:03.296930 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:03.297081 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:03.297456 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:03.297461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:03.297473 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:03.298072 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:03.298385 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:03.298794 (+ 409us) server_negotiation.cc:299] Negotiation successful
0621 06:52:03.299206 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":344,"threads_started":1}
I20260621 06:52:03.303032 23437 raft_consensus.cc:348] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b201a2367954475c94b63757627835dd" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45361 } }
I20260621 06:52:03.303390 23437 raft_consensus.cc:374] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:03.303491 23437 raft_consensus.cc:729] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b201a2367954475c94b63757627835dd, State: Initialized, Role: FOLLOWER
I20260621 06:52:03.303989 23437 consensus_queue.cc:260] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b201a2367954475c94b63757627835dd" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45361 } }
I20260621 06:52:03.304246 23437 raft_consensus.cc:388] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:03.304391 23437 raft_consensus.cc:482] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:03.304601 23437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:03.306681 23437 raft_consensus.cc:504] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b201a2367954475c94b63757627835dd" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45361 } }
I20260621 06:52:03.307190 23437 leader_election.cc:302] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b201a2367954475c94b63757627835dd; no voters:
I20260621 06:52:03.307755 23437 leader_election.cc:288] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:03.307870 23442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:03.309118 23442 raft_consensus.cc:686] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [term 1 LEADER]: Becoming Leader. State: Replica: b201a2367954475c94b63757627835dd, State: Running, Role: LEADER
I20260621 06:52:03.309697 23442 consensus_queue.cc:237] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b201a2367954475c94b63757627835dd" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45361 } }
I20260621 06:52:03.309819 23437 sys_catalog.cc:564] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:03.312616 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b201a2367954475c94b63757627835dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b201a2367954475c94b63757627835dd" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45361 } } }
I20260621 06:52:03.313030 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:03.312589 23444 sys_catalog.cc:455] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b201a2367954475c94b63757627835dd. Latest consensus state: current_term: 1 leader_uuid: "b201a2367954475c94b63757627835dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b201a2367954475c94b63757627835dd" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45361 } } }
I20260621 06:52:03.313556 23444 sys_catalog.cc:458] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:03.317934 23449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:03.323035 23449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:03.331121 23449 catalog_manager.cc:1347] Generated new cluster ID: bfa0777a5b5941678bf5578706a23701
I20260621 06:52:03.331240 23449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:03.340934 23449 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:03.342687 23449 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:03.352936 23449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b201a2367954475c94b63757627835dd: Generated new TSK 0
I20260621 06:52:03.353961 23449 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:52:03.399312 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45361
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:03.715530 23463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:03.715802 23463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:03.715960 23463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:03.723181 23463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:03.723276 23463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:03.723380 23463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:52:03.732908 23463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:03.734196 23463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:03.735492 23463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:03.747952 23468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:03.747988 23469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:03.747918 23471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:03.749119 23463 server_base.cc:1034] running on GCE node
I20260621 06:52:03.749691 23463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:03.750625 23463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:03.751844 23463 hybrid_clock.cc:648] HybridClock initialized: now 1782024723751807 us; error 48 us; skew 500 ppm
I20260621 06:52:03.752133 23463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:03 dist-test-slave-gmf2 krb5kdc[23357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024723, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:03.755074 23463 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:52:03.756497 23463 webserver.cc:460] Webserver started at http://127.20.120.193:32947/ using document root <none> and password file <none>
I20260621 06:52:03.757156 23463 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:03.757277 23463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:03.757572 23463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:03.760320 23463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "f2e160024fae421288715cf74373cf36"
format_stamp: "Formatted at 2026-06-21 06:52:03 on dist-test-slave-gmf2"
server_key: "8c04ed2e418fc1febcce8ca042ef8409"
server_key_iv: "81b6cbd2a801d0ed7c06a10304d97d06"
server_key_version: "encryptionkey@0"
I20260621 06:52:03.761085 23463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "f2e160024fae421288715cf74373cf36"
format_stamp: "Formatted at 2026-06-21 06:52:03 on dist-test-slave-gmf2"
server_key: "8c04ed2e418fc1febcce8ca042ef8409"
server_key_iv: "81b6cbd2a801d0ed7c06a10304d97d06"
server_key_version: "encryptionkey@0"
I20260621 06:52:03.766167 23463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 06:52:03.769454 23478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:03.770789 23463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:03.770955 23463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "f2e160024fae421288715cf74373cf36"
format_stamp: "Formatted at 2026-06-21 06:52:03 on dist-test-slave-gmf2"
server_key: "8c04ed2e418fc1febcce8ca042ef8409"
server_key_iv: "81b6cbd2a801d0ed7c06a10304d97d06"
server_key_version: "encryptionkey@0"
I20260621 06:52:03.771147 23463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:03.784940 23463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:03.788831 23463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:03.789122 23463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:03.790117 23463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:03.791318 23463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:03.791388 23463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:03.791540 23463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:03.791601 23463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:03.823108 23463 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:34297
I20260621 06:52:03.823194 23590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:34297 every 8 connection(s)
I20260621 06:52:03.824345 23463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:52:03.827121 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 23463
I20260621 06:52:03.827337 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:52:03.827976 20963 external_mini_cluster.cc:1428] Setting key a62ec7046ba5ebd496e4a68a68c5ae23
Jun 21 06:52:03 dist-test-slave-gmf2 krb5kdc[23357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024723, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:03.845168 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:03.826755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:35075 (local address 127.20.120.254:45361)
0621 06:52:03.827235 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:03.827243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:03.828407 (+ 1164us) server_negotiation.cc:407] Connection header received
0621 06:52:03.830255 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:03.830260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:03.830338 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:03.830460 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:03.831855 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:03.832982 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:03.834533 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:03.834922 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:03.835052 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:03.838517 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:03.838526 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:03.838535 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:03.838578 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:03.842026 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:03.842694 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:03.842703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:03.842709 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:03.842862 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:03.843297 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:03.843318 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:03.843323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:03.843749 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:03.843904 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:03.844545 (+ 641us) server_negotiation.cc:299] Negotiation successful
0621 06:52:03.844768 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":231,"threads_started":1}
I20260621 06:52:03.845861 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:03.827397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45361 (local address 127.20.120.193:35075)
0621 06:52:03.828220 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:03.828267 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:03.830010 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:03.830664 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:03.830672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:03.831150 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:03.831691 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:03.831707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:03.833130 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:03.833134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:03.834388 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:03.834398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:03.835131 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:03.835138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:03.835373 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:03.836047 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:03.836068 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:03.838295 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:03.842165 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:03.842171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:03.842183 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:03.842557 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:03.842965 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:03.842968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:03.842970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:03.843166 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:03.843866 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:03.843872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:03.844168 (+ 296us) client_negotiation.cc:769] Sending connection context
0621 06:52:03.844382 (+ 214us) client_negotiation.cc:240] Negotiation successful
0621 06:52:03.844733 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":326,"threads_started":1}
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
I20260621 06:52:03.847582 23591 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45361
I20260621 06:52:03.847862 23591 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:03.848528 23591 heartbeater.cc:511] Master 127.20.120.254:45361 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
I20260621 06:52:03.850838 23402 ts_manager.cc:194] Registered new tserver with Master: f2e160024fae421288715cf74373cf36 (127.20.120.193:34297)
I20260621 06:52:03.853127 23402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:35075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:52:03.872881 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45361
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:04.190707 23597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:04.191008 23597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:04.191185 23597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:04.198207 23597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:04.198356 23597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:04.198539 23597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:52:04.207877 23597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:04.209141 23597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:04.210516 23597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:04.223580 23602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:04.223623 23605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:04.223645 23603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:04.224406 23597 server_base.cc:1034] running on GCE node
I20260621 06:52:04.224941 23597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:04.225837 23597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:04.227022 23597 hybrid_clock.cc:648] HybridClock initialized: now 1782024724226970 us; error 51 us; skew 500 ppm
I20260621 06:52:04.227350 23597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:04 dist-test-slave-gmf2 krb5kdc[23357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024724, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:04.230335 23597 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:52:04.231616 23597 webserver.cc:460] Webserver started at http://127.20.120.194:43183/ using document root <none> and password file <none>
I20260621 06:52:04.232281 23597 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:04.232409 23597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:04.232712 23597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:04.235590 23597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "4828c3efd43e4ece98d4fe2ed42c0bcb"
format_stamp: "Formatted at 2026-06-21 06:52:04 on dist-test-slave-gmf2"
server_key: "375ac60459fe3a8c9d85d904b218d40f"
server_key_iv: "6782c35424058a92c02e67583c7fa89d"
server_key_version: "encryptionkey@0"
I20260621 06:52:04.236351 23597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "4828c3efd43e4ece98d4fe2ed42c0bcb"
format_stamp: "Formatted at 2026-06-21 06:52:04 on dist-test-slave-gmf2"
server_key: "375ac60459fe3a8c9d85d904b218d40f"
server_key_iv: "6782c35424058a92c02e67583c7fa89d"
server_key_version: "encryptionkey@0"
I20260621 06:52:04.241078 23597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 06:52:04.244273 23612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:04.245469 23597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 06:52:04.245630 23597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "4828c3efd43e4ece98d4fe2ed42c0bcb"
format_stamp: "Formatted at 2026-06-21 06:52:04 on dist-test-slave-gmf2"
server_key: "375ac60459fe3a8c9d85d904b218d40f"
server_key_iv: "6782c35424058a92c02e67583c7fa89d"
server_key_version: "encryptionkey@0"
I20260621 06:52:04.245805 23597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:04.259497 23597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:04.263041 23597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:04.263311 23597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:04.264125 23597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:04.265419 23597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:04.265488 23597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:04.265664 23597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:04.265714 23597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:04.314028 23597 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:41143
I20260621 06:52:04.314070 23724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:41143 every 8 connection(s)
I20260621 06:52:04.315464 23597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:52:04.319499 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 23597
I20260621 06:52:04.319705 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:52:04.320094 20963 external_mini_cluster.cc:1428] Setting key 1d70ec2e73d410a6b7aff32e9832fe25
Jun 21 06:52:04 dist-test-slave-gmf2 krb5kdc[23357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024724, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:04.334996 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:04.317894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:43201 (local address 127.20.120.254:45361)
0621 06:52:04.318103 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:04.318107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:04.319219 (+ 1112us) server_negotiation.cc:407] Connection header received
0621 06:52:04.321213 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:04.321218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:04.321308 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:04.321418 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:04.322977 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:04.323660 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:04.325128 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:04.325541 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:04.325677 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:04.329625 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:04.329630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:04.329634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:04.329664 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:04.332197 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:04.332901 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:04.332906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:04.332916 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:04.333032 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:04.333430 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:04.333434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:04.333437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:04.333674 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:04.333826 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:04.334512 (+ 686us) server_negotiation.cc:299] Negotiation successful
0621 06:52:04.334704 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 06:52:04.335966 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:04.318411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45361 (local address 127.20.120.194:43201)
0621 06:52:04.319074 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:04.319111 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:04.320956 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:04.321686 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:04.321696 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:04.322310 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:04.322822 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:04.322838 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:04.323835 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:04.323839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:04.324982 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:04.324993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:04.325769 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:04.325776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:04.326071 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:04.326818 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:04.326854 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:04.329423 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:04.332332 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:04.332341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:04.332355 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:04.332763 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:04.333135 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:04.333141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:04.333145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:04.333314 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:04.333799 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:04.333806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:04.334174 (+ 368us) client_negotiation.cc:769] Sending connection context
0621 06:52:04.334415 (+ 241us) client_negotiation.cc:240] Negotiation successful
0621 06:52:04.334873 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":251,"threads_started":1}
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
I20260621 06:52:04.337373 23725 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45361
I20260621 06:52:04.337698 23725 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:04.338474 23725 heartbeater.cc:511] Master 127.20.120.254:45361 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
I20260621 06:52:04.339922 23402 ts_manager.cc:194] Registered new tserver with Master: 4828c3efd43e4ece98d4fe2ed42c0bcb (127.20.120.194:41143)
I20260621 06:52:04.340703 23402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:43201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:52:04.360108 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45361
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:04.661213 23730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:04.661541 23730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:04.661648 23730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:04.668596 23730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:04.668689 23730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:04.668839 23730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:52:04.677977 23730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:04.679201 23730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:04.680420 23730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:04.692617 23735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:04.692692 23738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:04.692721 23736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:04.693454 23730 server_base.cc:1034] running on GCE node
I20260621 06:52:04.694115 23730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:04.695044 23730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:04.696259 23730 hybrid_clock.cc:648] HybridClock initialized: now 1782024724696220 us; error 35 us; skew 500 ppm
I20260621 06:52:04.696611 23730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:04 dist-test-slave-gmf2 krb5kdc[23357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024724, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:04.704162 23730 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:52:04.705562 23730 webserver.cc:460] Webserver started at http://127.20.120.195:39271/ using document root <none> and password file <none>
I20260621 06:52:04.706300 23730 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:04.706425 23730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:04.706745 23730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:04.709583 23730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "f85cfed455354e84ad366b931722f5bd"
format_stamp: "Formatted at 2026-06-21 06:52:04 on dist-test-slave-gmf2"
server_key: "edb5c2b67c1cb66b581c152a552a4536"
server_key_iv: "7b44796aeb6bfd9fae7202be39e2205b"
server_key_version: "encryptionkey@0"
I20260621 06:52:04.710387 23730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "f85cfed455354e84ad366b931722f5bd"
format_stamp: "Formatted at 2026-06-21 06:52:04 on dist-test-slave-gmf2"
server_key: "edb5c2b67c1cb66b581c152a552a4536"
server_key_iv: "7b44796aeb6bfd9fae7202be39e2205b"
server_key_version: "encryptionkey@0"
I20260621 06:52:04.715538 23730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 06:52:04.718778 23745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:04.719990 23730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:52:04.720146 23730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "f85cfed455354e84ad366b931722f5bd"
format_stamp: "Formatted at 2026-06-21 06:52:04 on dist-test-slave-gmf2"
server_key: "edb5c2b67c1cb66b581c152a552a4536"
server_key_iv: "7b44796aeb6bfd9fae7202be39e2205b"
server_key_version: "encryptionkey@0"
I20260621 06:52:04.720329 23730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:04.733501 23730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:04.737331 23730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:04.737617 23730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:04.738539 23730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:04.739742 23730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:04.739843 23730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:04.739987 23730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:04.740062 23730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:04.772004 23730 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:46371
I20260621 06:52:04.772045 23857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:46371 every 8 connection(s)
I20260621 06:52:04.773270 23730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:52:04.776160 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 23730
I20260621 06:52:04.776337 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:52:04.776923 20963 external_mini_cluster.cc:1428] Setting key c79fe89c56369c4172363f007f006f1c
Jun 21 06:52:04 dist-test-slave-gmf2 krb5kdc[23357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024724, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:04.792083 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:04.775732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:41831 (local address 127.20.120.254:45361)
0621 06:52:04.775926 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:04.775932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:04.777365 (+ 1433us) server_negotiation.cc:407] Connection header received
0621 06:52:04.779447 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:04.779452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:04.779534 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:04.779643 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:04.781052 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:04.781799 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:04.783273 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:04.783635 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:04.783763 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:04.786981 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:04.786987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:04.786990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:04.787019 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:04.789262 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:04.789946 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:04.789953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:04.789961 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:04.790072 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:04.790480 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:04.790485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:04.790488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:04.790763 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:04.790909 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:04.791513 (+ 604us) server_negotiation.cc:299] Negotiation successful
0621 06:52:04.791762 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 06:52:04.792917 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:04.776431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45361 (local address 127.20.120.195:41831)
0621 06:52:04.777217 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:04.777259 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:04.779207 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:04.779865 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:04.779873 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:04.780353 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:04.780903 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:04.780918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:04.781960 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:04.781971 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:04.783115 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:04.783126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:04.783837 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:04.783846 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:04.784058 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:04.784733 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:04.784752 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:04.786789 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:04.789405 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:04.789413 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:04.789428 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:04.789804 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:04.790188 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:04.790193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:04.790196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:04.790362 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:04.790883 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:04.790890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:04.791190 (+ 300us) client_negotiation.cc:769] Sending connection context
0621 06:52:04.791420 (+ 230us) client_negotiation.cc:240] Negotiation successful
0621 06:52:04.791904 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":308,"threads_started":1}
I20260621 06:52:04.794127 23858 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45361
I20260621 06:52:04.794422 23858 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:04.795082 23858 heartbeater.cc:511] Master 127.20.120.254:45361 requested a full tablet report, sending...
I20260621 06:52:04.796391 23402 ts_manager.cc:194] Registered new tserver with Master: f85cfed455354e84ad366b931722f5bd (127.20.120.195:46371)
I20260621 06:52:04.797148 23402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:41831
I20260621 06:52:04.803714 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 06:52:04.831957 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:04.829650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49436 (local address 127.20.120.254:45361)
0621 06:52:04.829855 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:04.829864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:04.830290 (+ 426us) server_negotiation.cc:407] Connection header received
0621 06:52:04.830495 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:04.830498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:04.830566 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:04.830664 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:04.831772 (+ 1108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49436: BlockingRecv error: recv got EOF from 127.0.0.1:49436 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 06:52:04.835006 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 23463
W20260621 06:52:04.837651 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 23463: Not authorized: Client connection negotiation failed: client connection to 127.20.120.193:34297: server requires authentication, but client does not have Kerberos credentials available
I20260621 06:52:04.837906 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 23463
I20260621 06:52:04.853513 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 23597
W20260621 06:52:04.855960 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 23597: Not authorized: Client connection negotiation failed: client connection to 127.20.120.194:41143: server requires authentication, but client does not have Kerberos credentials available
I20260621 06:52:04.856127 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 23597
I20260621 06:52:04.871308 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 23730
W20260621 06:52:04.873857 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 23730: Not authorized: Client connection negotiation failed: client connection to 127.20.120.195:46371: server requires authentication, but client does not have Kerberos credentials available
I20260621 06:52:04.874027 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 23730
I20260621 06:52:04.888499 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 23370
I20260621 06:52:04.889580 23432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:04.994056 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 23370
2026-06-21T06:52:05Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3367 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:52:05 dist-test-slave-gmf2 krb5kdc[23875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:52:05 dist-test-slave-gmf2 krb5kdc[23875](info): set up 2 sockets
Jun 21 06:52:05 dist-test-slave-gmf2 krb5kdc[23875](info): commencing operation
krb5kdc: starting...
W20260621 06:52:06.050335 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:52:06 dist-test-slave-gmf2 krb5kdc[23875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:52:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:52:06.232962 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:45863
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:45863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:06.546530 23888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:06.546811 23888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:06.546897 23888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:06.554072 23888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:06.554176 23888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:06.554222 23888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:06.554281 23888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:06.554327 23888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:06.563936 23888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:45863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:45863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:06.565106 23888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:06.566414 23888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:06.578003 23893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:06.578038 23894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:06.577960 23896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:06.578847 23888 server_base.cc:1034] running on GCE node
I20260621 06:52:06.579612 23888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:06.580983 23888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:06.582207 23888 hybrid_clock.cc:648] HybridClock initialized: now 1782024726582170 us; error 53 us; skew 500 ppm
I20260621 06:52:06.582548 23888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:06 dist-test-slave-gmf2 krb5kdc[23875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024726, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:06.585547 23888 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:52:06.587037 23888 webserver.cc:460] Webserver started at http://127.20.120.254:34517/ using document root <none> and password file <none>
I20260621 06:52:06.587721 23888 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:06.587864 23888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:06.588178 23888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:06.591140 23888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "5a10213c50374cd289221b3be8581de5"
format_stamp: "Formatted at 2026-06-21 06:52:06 on dist-test-slave-gmf2"
server_key: "43bc1c6e368c268329a32e86fc3a86a5"
server_key_iv: "25b4703e82e17607493ee7cf74bae230"
server_key_version: "encryptionkey@0"
I20260621 06:52:06.591928 23888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "5a10213c50374cd289221b3be8581de5"
format_stamp: "Formatted at 2026-06-21 06:52:06 on dist-test-slave-gmf2"
server_key: "43bc1c6e368c268329a32e86fc3a86a5"
server_key_iv: "25b4703e82e17607493ee7cf74bae230"
server_key_version: "encryptionkey@0"
I20260621 06:52:06.597298 23888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260621 06:52:06.600708 23903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:06.602100 23888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 06:52:06.602278 23888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "5a10213c50374cd289221b3be8581de5"
format_stamp: "Formatted at 2026-06-21 06:52:06 on dist-test-slave-gmf2"
server_key: "43bc1c6e368c268329a32e86fc3a86a5"
server_key_iv: "25b4703e82e17607493ee7cf74bae230"
server_key_version: "encryptionkey@0"
I20260621 06:52:06.602475 23888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:06.622459 23888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:06.626240 23888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:06.626504 23888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:06.644657 23888 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:45863
I20260621 06:52:06.644702 23954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:45863 every 8 connection(s)
I20260621 06:52:06.646014 23888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:06.649518 23955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:06.650300 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 23888
I20260621 06:52:06.650512 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:06.651139 20963 external_mini_cluster.cc:1428] Setting key 699636441ca60ca9038904acd610ac8f
I20260621 06:52:06.661389 23955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5: Bootstrap starting.
Jun 21 06:52:06 dist-test-slave-gmf2 krb5kdc[23875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:06.664505 23955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:06.665709 23955 log.cc:824] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:06.668937 23955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5: No bootstrap required, opened a new log
I20260621 06:52:06.673391 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:06.652985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.20.120.254:45863)
0621 06:52:06.653725 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:06.653740 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:06.653777 (+ 37us) server_negotiation.cc:407] Connection header received
0621 06:52:06.655551 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:06.655570 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:06.655976 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:06.656353 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:06.657803 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:06.659123 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:06.660350 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:06.660973 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:06.661159 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:06.665416 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:06.665423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:06.665439 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:06.665469 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:06.669167 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:06.669783 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:06.669791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:06.669798 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:06.670028 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:06.670365 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:06.670370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:06.670374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:06.671062 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:06.671334 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:06.671622 (+ 288us) server_negotiation.cc:299] Negotiation successful
0621 06:52:06.672081 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":331,"threads_started":1}
I20260621 06:52:06.677076 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:52:06.679023 23955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a10213c50374cd289221b3be8581de5" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45863 } }
I20260621 06:52:06.679463 23955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:06.679576 23955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a10213c50374cd289221b3be8581de5, State: Initialized, Role: FOLLOWER
I20260621 06:52:06.680236 23955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a10213c50374cd289221b3be8581de5" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45863 } }
I20260621 06:52:06.680545 23955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:06.680698 23955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:06.680896 23955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:06.684716 23955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a10213c50374cd289221b3be8581de5" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45863 } }
I20260621 06:52:06.685312 23955 leader_election.cc:302] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a10213c50374cd289221b3be8581de5; no voters:
I20260621 06:52:06.686026 23955 leader_election.cc:288] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:06.686139 23961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:06.687465 23961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [term 1 LEADER]: Becoming Leader. State: Replica: 5a10213c50374cd289221b3be8581de5, State: Running, Role: LEADER
I20260621 06:52:06.688104 23961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a10213c50374cd289221b3be8581de5" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45863 } }
I20260621 06:52:06.688539 23955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:06.691092 23963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a10213c50374cd289221b3be8581de5. Latest consensus state: current_term: 1 leader_uuid: "5a10213c50374cd289221b3be8581de5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a10213c50374cd289221b3be8581de5" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45863 } } }
I20260621 06:52:06.691816 23962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a10213c50374cd289221b3be8581de5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a10213c50374cd289221b3be8581de5" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45863 } } }
I20260621 06:52:06.692297 23962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:06.692423 23963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:06.702623 23969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:06.707638 23969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:06.716928 23969 catalog_manager.cc:1347] Generated new cluster ID: 65383aa80d3843acb2e59384642336d1
I20260621 06:52:06.717038 23969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:06.735121 23969 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:06.737031 23969 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:06.744136 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:06.735426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.20.120.254:45863)
0621 06:52:06.735629 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:06.735638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:06.736586 (+ 948us) server_negotiation.cc:407] Connection header received
0621 06:52:06.736835 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:06.736841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:06.737051 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:06.737198 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:06.738463 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:06.739287 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:06.740676 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:06.741127 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:06.741273 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:06.742536 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:06.742541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:06.742545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:06.743211 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:06.743276 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:06.743302 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:06.743433 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:06.743604 (+ 171us) server_negotiation.cc:299] Negotiation successful
0621 06:52:06.743712 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 06:52:06.749235 23969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a10213c50374cd289221b3be8581de5: Generated new TSK 0
I20260621 06:52:06.750243 23969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:52:06.757145 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 23888
I20260621 06:52:06.758337 23950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:06.844134 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 23888
2026-06-21T06:52:06Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1845 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:52:06 dist-test-slave-gmf2 krb5kdc[23989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:52:06 dist-test-slave-gmf2 krb5kdc[23989](info): set up 2 sockets
Jun 21 06:52:06 dist-test-slave-gmf2 krb5kdc[23989](info): commencing operation
krb5kdc: starting...
W20260621 06:52:07.914572 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:52:08 dist-test-slave-gmf2 krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:52:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:52:08.100566 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:46569
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:46569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:08.421571 24002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:08.421979 24002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:08.422150 24002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:08.429567 24002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:08.429729 24002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:08.429850 24002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:08.429996 24002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:08.430081 24002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:08.440088 24002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:46569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:46569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:08.441501 24002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:08.442901 24002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:08.454877 24007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:08.454952 24010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:08.455006 24008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:08.455747 24002 server_base.cc:1034] running on GCE node
I20260621 06:52:08.456506 24002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:08.458015 24002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:08.459230 24002 hybrid_clock.cc:648] HybridClock initialized: now 1782024728459190 us; error 35 us; skew 500 ppm
I20260621 06:52:08.459545 24002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:08 dist-test-slave-gmf2 krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024728, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:08.462641 24002 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:52:08.464102 24002 webserver.cc:460] Webserver started at http://127.20.120.254:39973/ using document root <none> and password file <none>
I20260621 06:52:08.464732 24002 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:08.464828 24002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:08.465080 24002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:08.468067 24002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "bf4c2a29950e4aee84bca0fd56776aed"
format_stamp: "Formatted at 2026-06-21 06:52:08 on dist-test-slave-gmf2"
server_key: "d7a94930153c0fcc501b28e520f43e3a"
server_key_iv: "5e936c251acbc88ceed3699f0953320a"
server_key_version: "encryptionkey@0"
I20260621 06:52:08.468780 24002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "bf4c2a29950e4aee84bca0fd56776aed"
format_stamp: "Formatted at 2026-06-21 06:52:08 on dist-test-slave-gmf2"
server_key: "d7a94930153c0fcc501b28e520f43e3a"
server_key_iv: "5e936c251acbc88ceed3699f0953320a"
server_key_version: "encryptionkey@0"
I20260621 06:52:08.473716 24002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 06:52:08.477031 24017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:08.478354 24002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:08.478524 24002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "bf4c2a29950e4aee84bca0fd56776aed"
format_stamp: "Formatted at 2026-06-21 06:52:08 on dist-test-slave-gmf2"
server_key: "d7a94930153c0fcc501b28e520f43e3a"
server_key_iv: "5e936c251acbc88ceed3699f0953320a"
server_key_version: "encryptionkey@0"
I20260621 06:52:08.478735 24002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:08.495558 24002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:08.499727 24002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:08.500010 24002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:08.517822 24002 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:46569
I20260621 06:52:08.517860 24068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:46569 every 8 connection(s)
I20260621 06:52:08.519238 24002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:08.523088 24069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:08.528033 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 24002
I20260621 06:52:08.528270 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:08.528899 20963 external_mini_cluster.cc:1428] Setting key fd83631a3f1625e67a3102cf0ade1410
I20260621 06:52:08.535171 24069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed: Bootstrap starting.
I20260621 06:52:08.539007 24069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:08.540083 24069 log.cc:824] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:08.543089 24069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed: No bootstrap required, opened a new log
Jun 21 06:52:08 dist-test-slave-gmf2 krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:08.553668 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:08.530956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.20.120.254:46569)
0621 06:52:08.531830 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:08.531846 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:08.531902 (+ 56us) server_negotiation.cc:407] Connection header received
0621 06:52:08.534258 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:08.534301 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:08.534820 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:08.535288 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:08.536569 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:08.538615 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:08.539889 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:08.540917 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:08.541049 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:08.545076 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:08.545086 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:08.545110 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:08.545161 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:08.549193 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:08.549852 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:08.549861 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:08.549868 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:08.550148 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:08.550505 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:08.550510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:08.550514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:08.551224 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:08.551537 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:08.551836 (+ 299us) server_negotiation.cc:299] Negotiation successful
0621 06:52:08.552308 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":345,"threads_started":1}
I20260621 06:52:08.553565 24069 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4c2a29950e4aee84bca0fd56776aed" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46569 } }
I20260621 06:52:08.554095 24069 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:08.554200 24069 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4c2a29950e4aee84bca0fd56776aed, State: Initialized, Role: FOLLOWER
I20260621 06:52:08.554838 24069 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4c2a29950e4aee84bca0fd56776aed" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46569 } }
I20260621 06:52:08.555058 24069 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:08.555197 24069 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:08.555397 24069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:08.558161 24069 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4c2a29950e4aee84bca0fd56776aed" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46569 } }
I20260621 06:52:08.558663 24069 leader_election.cc:302] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf4c2a29950e4aee84bca0fd56776aed; no voters:
I20260621 06:52:08.559345 24069 leader_election.cc:288] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:08.559449 24074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:08.560072 24074 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [term 1 LEADER]: Becoming Leader. State: Replica: bf4c2a29950e4aee84bca0fd56776aed, State: Running, Role: LEADER
I20260621 06:52:08.560580 24074 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4c2a29950e4aee84bca0fd56776aed" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46569 } }
I20260621 06:52:08.561517 24069 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:08.563813 24075 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf4c2a29950e4aee84bca0fd56776aed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4c2a29950e4aee84bca0fd56776aed" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46569 } } }
I20260621 06:52:08.564175 24075 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:08.564877 24076 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4c2a29950e4aee84bca0fd56776aed. Latest consensus state: current_term: 1 leader_uuid: "bf4c2a29950e4aee84bca0fd56776aed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4c2a29950e4aee84bca0fd56776aed" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46569 } } }
I20260621 06:52:08.565124 24081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:08.565169 24076 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:08.572166 24081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:08.583197 24081 catalog_manager.cc:1347] Generated new cluster ID: d18e30d37eb8400f8071edaf2d7159ce
I20260621 06:52:08.583339 24081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:08.602473 24081 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:08.603907 24081 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:08.613976 24081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf4c2a29950e4aee84bca0fd56776aed: Generated new TSK 0
I20260621 06:52:08.614849 24081 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:52:08.673985 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:46569
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:09.000882 24095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:09.001204 24095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:09.001360 24095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:09.008687 24095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:09.008832 24095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:09.008997 24095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:52:09.018503 24095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:46569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:09.019778 24095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:09.021090 24095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:09.034143 24100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:09.034173 24103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:09.034107 24101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:09.034986 24095 server_base.cc:1034] running on GCE node
I20260621 06:52:09.035529 24095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:09.036422 24095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:09.037609 24095 hybrid_clock.cc:648] HybridClock initialized: now 1782024729037556 us; error 51 us; skew 500 ppm
I20260621 06:52:09.037947 24095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:09 dist-test-slave-gmf2 krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024729, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:09.040998 24095 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:52:09.042425 24095 webserver.cc:460] Webserver started at http://127.20.120.193:39271/ using document root <none> and password file <none>
I20260621 06:52:09.043042 24095 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:09.043135 24095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:09.043375 24095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:09.046167 24095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "5f633e871a1240999cc9169a6292c125"
format_stamp: "Formatted at 2026-06-21 06:52:09 on dist-test-slave-gmf2"
server_key: "625b206609e5e7fb3e06dde72da7d607"
server_key_iv: "05a0d5220c515abb696d53f1f5e70001"
server_key_version: "encryptionkey@0"
I20260621 06:52:09.046876 24095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "5f633e871a1240999cc9169a6292c125"
format_stamp: "Formatted at 2026-06-21 06:52:09 on dist-test-slave-gmf2"
server_key: "625b206609e5e7fb3e06dde72da7d607"
server_key_iv: "05a0d5220c515abb696d53f1f5e70001"
server_key_version: "encryptionkey@0"
I20260621 06:52:09.051646 24095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260621 06:52:09.054869 24110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:09.056237 24095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:52:09.056398 24095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "5f633e871a1240999cc9169a6292c125"
format_stamp: "Formatted at 2026-06-21 06:52:09 on dist-test-slave-gmf2"
server_key: "625b206609e5e7fb3e06dde72da7d607"
server_key_iv: "05a0d5220c515abb696d53f1f5e70001"
server_key_version: "encryptionkey@0"
I20260621 06:52:09.056582 24095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:09.072402 24095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:09.076531 24095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:09.076817 24095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:09.077697 24095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:09.078866 24095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:09.078985 24095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:09.079149 24095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:09.079221 24095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:09.111301 24095 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:44285
I20260621 06:52:09.111341 24222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:44285 every 8 connection(s)
I20260621 06:52:09.112588 24095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:52:09.114373 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 24095
I20260621 06:52:09.114571 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:52:09.115298 20963 external_mini_cluster.cc:1428] Setting key 48710a4c23cfcdd1142cf7cd078dfc2d
Jun 21 06:52:09 dist-test-slave-gmf2 krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024729, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
I20260621 06:52:09.131435 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:09.115304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:47365 (local address 127.20.120.254:46569)
0621 06:52:09.115750 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:09.115756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:09.117984 (+ 2228us) server_negotiation.cc:407] Connection header received
0621 06:52:09.118744 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:09.118749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:09.118840 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:09.118950 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:09.120285 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:09.120931 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:09.122428 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:09.122803 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:09.122931 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:09.126266 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:09.126271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:09.126274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:09.126301 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:09.128691 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:09.129307 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:09.129316 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:09.129318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:09.129416 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:09.129815 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:09.129819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:09.129823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:09.130127 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:09.130269 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:09.130917 (+ 648us) server_negotiation.cc:299] Negotiation successful
0621 06:52:09.131116 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":214,"threads_started":1}
I20260621 06:52:09.132375 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:09.115849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:46569 (local address 127.20.120.193:47365)
0621 06:52:09.116735 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:09.116783 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:09.118482 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:09.119173 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:09.119180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:09.119640 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:09.120138 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:09.120153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:09.121076 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:09.121083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:09.122284 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:09.122295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:09.123008 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:09.123016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:09.123239 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:09.123892 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:09.123912 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:09.126073 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:09.128825 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:09.128832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:09.128845 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:09.129192 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:09.129514 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:09.129520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:09.129523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:09.129686 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:09.130254 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:09.130261 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:09.130574 (+ 313us) client_negotiation.cc:769] Sending connection context
0621 06:52:09.130812 (+ 238us) client_negotiation.cc:240] Negotiation successful
0621 06:52:09.131253 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":373,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
I20260621 06:52:09.133771 24223 heartbeater.cc:347] Connected to a master server at 127.20.120.254:46569
I20260621 06:52:09.134074 24223 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:09.134764 24223 heartbeater.cc:511] Master 127.20.120.254:46569 requested a full tablet report, sending...
I20260621 06:52:09.136698 24034 ts_manager.cc:194] Registered new tserver with Master: 5f633e871a1240999cc9169a6292c125 (127.20.120.193:44285)
I20260621 06:52:09.138149 24034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:47365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:52:09.154188 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:46569
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:09.464166 24229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:09.464408 24229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:09.464584 24229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:09.471544 24229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:09.471659 24229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:09.471845 24229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:52:09.481226 24229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:46569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:09.482463 24229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:09.483732 24229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:09.495982 24234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:09.496033 24237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:09.496063 24235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:09.496820 24229 server_base.cc:1034] running on GCE node
I20260621 06:52:09.497377 24229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:09.498311 24229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:09.499532 24229 hybrid_clock.cc:648] HybridClock initialized: now 1782024729499485 us; error 31 us; skew 500 ppm
I20260621 06:52:09.499848 24229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:09 dist-test-slave-gmf2 krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024729, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:09.502902 24229 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:52:09.504334 24229 webserver.cc:460] Webserver started at http://127.20.120.194:44663/ using document root <none> and password file <none>
I20260621 06:52:09.504961 24229 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:09.505096 24229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:09.505396 24229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:09.508174 24229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "1272f067416a40eca99da5cedf1b8a9f"
format_stamp: "Formatted at 2026-06-21 06:52:09 on dist-test-slave-gmf2"
server_key: "047a5d704dc5734613876bd539f4b56e"
server_key_iv: "1158d3a894c589d681e6f5f340f12e20"
server_key_version: "encryptionkey@0"
I20260621 06:52:09.508911 24229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "1272f067416a40eca99da5cedf1b8a9f"
format_stamp: "Formatted at 2026-06-21 06:52:09 on dist-test-slave-gmf2"
server_key: "047a5d704dc5734613876bd539f4b56e"
server_key_iv: "1158d3a894c589d681e6f5f340f12e20"
server_key_version: "encryptionkey@0"
I20260621 06:52:09.513805 24229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260621 06:52:09.517033 24244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:09.518653 24229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:52:09.518821 24229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "1272f067416a40eca99da5cedf1b8a9f"
format_stamp: "Formatted at 2026-06-21 06:52:09 on dist-test-slave-gmf2"
server_key: "047a5d704dc5734613876bd539f4b56e"
server_key_iv: "1158d3a894c589d681e6f5f340f12e20"
server_key_version: "encryptionkey@0"
I20260621 06:52:09.519032 24229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:09.551115 24229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:09.555254 24229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:09.555490 24229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:09.556355 24229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:09.557535 24229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:09.557603 24229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:09.557682 24229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:09.557744 24229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:09.589900 24229 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:43711
I20260621 06:52:09.590031 24356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:43711 every 8 connection(s)
I20260621 06:52:09.591277 24229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:52:09.601205 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 24229
I20260621 06:52:09.601390 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:52:09.601847 20963 external_mini_cluster.cc:1428] Setting key 2e50775a67ef596c39ad41ff13de9f44
Jun 21 06:52:09 dist-test-slave-gmf2 krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024729, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:09.611846 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:09.593668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:37977 (local address 127.20.120.254:46569)
0621 06:52:09.593847 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:09.593852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:09.595022 (+ 1170us) server_negotiation.cc:407] Connection header received
0621 06:52:09.596841 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:09.596847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:09.596944 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:09.597056 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:09.598516 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:09.599221 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:09.600749 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:09.601147 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:09.601278 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:09.605066 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:09.605075 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:09.605084 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:09.605130 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:09.608651 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:09.609342 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:09.609350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:09.609356 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:09.609507 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:09.609937 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:09.609943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:09.609948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:09.610308 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:09.610454 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:09.611121 (+ 667us) server_negotiation.cc:299] Negotiation successful
0621 06:52:09.611363 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 06:52:09.612433 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:09.594239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:46569 (local address 127.20.120.194:37977)
0621 06:52:09.594869 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:09.594912 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:09.596568 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:09.597278 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:09.597285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:09.597749 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:09.598322 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:09.598339 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:09.599364 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:09.599369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:09.600609 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:09.600621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:09.601367 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:09.601374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:09.601633 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:09.602369 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:09.602391 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:09.604843 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:09.608796 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:09.608802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:09.608815 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:09.609213 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:09.609619 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:09.609622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:09.609624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:09.609798 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:09.610437 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:09.610442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:09.610766 (+ 324us) client_negotiation.cc:769] Sending connection context
0621 06:52:09.610976 (+ 210us) client_negotiation.cc:240] Negotiation successful
0621 06:52:09.611345 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":258,"threads_started":1}
I20260621 06:52:09.614177 24357 heartbeater.cc:347] Connected to a master server at 127.20.120.254:46569
I20260621 06:52:09.614472 24357 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:09.615190 24357 heartbeater.cc:511] Master 127.20.120.254:46569 requested a full tablet report, sending...
I20260621 06:52:09.616593 24034 ts_manager.cc:194] Registered new tserver with Master: 1272f067416a40eca99da5cedf1b8a9f (127.20.120.194:43711)
I20260621 06:52:09.617406 24034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:37977
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:52:09.645628 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:46569
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:09.962332 24362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:09.962632 24362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:09.962791 24362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:09.970199 24362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:09.970304 24362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:09.970414 24362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:52:09.980369 24362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:46569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:09.981717 24362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:09.983285 24362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:09.996641 24370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:09.996685 24368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:09.996744 24367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:10.002544 24362 server_base.cc:1034] running on GCE node
I20260621 06:52:10.003194 24362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:10.004199 24362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:10.005419 24362 hybrid_clock.cc:648] HybridClock initialized: now 1782024730005346 us; error 59 us; skew 500 ppm
I20260621 06:52:10.005764 24362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:10 dist-test-slave-gmf2 krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024730, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:10.012158 24362 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:52:10.013700 24362 webserver.cc:460] Webserver started at http://127.20.120.195:34349/ using document root <none> and password file <none>
I20260621 06:52:10.014462 24362 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:10.014568 24362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:10.014819 24362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:10.017732 24362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "ba446f512bf84ae38a932ce00d8efea2"
format_stamp: "Formatted at 2026-06-21 06:52:10 on dist-test-slave-gmf2"
server_key: "16a03c55adef2a2ed7c31121ab1fda46"
server_key_iv: "f808827f4b5e7ea42b14e868505ff042"
server_key_version: "encryptionkey@0"
I20260621 06:52:10.018573 24362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "ba446f512bf84ae38a932ce00d8efea2"
format_stamp: "Formatted at 2026-06-21 06:52:10 on dist-test-slave-gmf2"
server_key: "16a03c55adef2a2ed7c31121ab1fda46"
server_key_iv: "f808827f4b5e7ea42b14e868505ff042"
server_key_version: "encryptionkey@0"
I20260621 06:52:10.023859 24362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 06:52:10.027423 24377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:10.028760 24362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:52:10.028893 24362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "ba446f512bf84ae38a932ce00d8efea2"
format_stamp: "Formatted at 2026-06-21 06:52:10 on dist-test-slave-gmf2"
server_key: "16a03c55adef2a2ed7c31121ab1fda46"
server_key_iv: "f808827f4b5e7ea42b14e868505ff042"
server_key_version: "encryptionkey@0"
I20260621 06:52:10.029078 24362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:10.043908 24362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:10.047605 24362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:10.047837 24362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:10.048699 24362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:10.049959 24362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:10.050040 24362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:10.050134 24362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:10.050180 24362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:10.082947 24362 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:33947
I20260621 06:52:10.082989 24489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:33947 every 8 connection(s)
I20260621 06:52:10.084286 24362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:52:10.092996 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 24362
I20260621 06:52:10.093200 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:52:10.093768 20963 external_mini_cluster.cc:1428] Setting key 3c8a167f87c50004fde93b0b8135f06c
Jun 21 06:52:10 dist-test-slave-gmf2 krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024730, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:10.104046 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:10.086780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:46017 (local address 127.20.120.254:46569)
0621 06:52:10.086975 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:10.086981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:10.088272 (+ 1291us) server_negotiation.cc:407] Connection header received
0621 06:52:10.090065 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:10.090069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:10.090149 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:10.090236 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:10.091649 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:10.092422 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:10.094650 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:10.095194 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:10.095321 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:10.098800 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:10.098806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:10.098817 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:10.098847 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:10.101151 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:10.101830 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:10.101835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:10.101838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:10.102023 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:10.102431 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:10.102436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:10.102439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:10.102707 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:10.102870 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:10.103497 (+ 627us) server_negotiation.cc:299] Negotiation successful
0621 06:52:10.103703 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 06:52:10.105049 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:10.087305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:46569 (local address 127.20.120.195:46017)
0621 06:52:10.088104 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:10.088143 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:10.089764 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:10.090449 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:10.090457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:10.090938 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:10.091483 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:10.091500 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:10.092595 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:10.092603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:10.094502 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:10.094513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:10.095415 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:10.095422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:10.095735 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:10.096467 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:10.096500 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:10.098607 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:10.101285 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:10.101293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:10.101309 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:10.101698 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:10.102128 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:10.102134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:10.102138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:10.102314 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:10.102845 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:10.102852 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:10.103173 (+ 321us) client_negotiation.cc:769] Sending connection context
0621 06:52:10.103420 (+ 247us) client_negotiation.cc:240] Negotiation successful
0621 06:52:10.103900 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":322,"threads_started":1}
I20260621 06:52:10.106379 24490 heartbeater.cc:347] Connected to a master server at 127.20.120.254:46569
I20260621 06:52:10.106647 24490 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:10.107293 24490 heartbeater.cc:511] Master 127.20.120.254:46569 requested a full tablet report, sending...
I20260621 06:52:10.108755 24033 ts_manager.cc:194] Registered new tserver with Master: ba446f512bf84ae38a932ce00d8efea2 (127.20.120.195:33947)
I20260621 06:52:10.109625 24033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:46017
I20260621 06:52:10.121024 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:52:10.140794 24223 heartbeater.cc:503] Master 127.20.120.254:46569 was elected leader, sending a full tablet report...
Jun 21 06:52:10 dist-test-slave-gmf2 krb5kdc[23989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024730, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 21 06:52:10 dist-test-slave-gmf2 krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024730, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:10.168879 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:10.154630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.20.120.254:46569)
0621 06:52:10.154819 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:10.154824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:10.155375 (+ 551us) server_negotiation.cc:407] Connection header received
0621 06:52:10.155592 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:10.155596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:10.155658 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:10.155769 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:10.157083 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:10.157806 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:10.159099 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:10.159529 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:10.159674 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:10.163706 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:10.163710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:10.163714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:10.163746 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:10.166359 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:10.166972 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:10.166984 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:10.166987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:10.167110 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:10.167501 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:10.167506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:10.167508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:10.167815 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:10.168063 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:10.168344 (+ 281us) server_negotiation.cc:299] Negotiation successful
0621 06:52:10.168551 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260621 06:52:10.169451 24033 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54102
I20260621 06:52:10.171159 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 24095
Jun 21 06:52:10 dist-test-slave-gmf2 krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024730, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:52:10.186429 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:10.172230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59074 (local address 127.20.120.193:44285)
0621 06:52:10.172844 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:10.172851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:10.172873 (+ 22us) server_negotiation.cc:407] Connection header received
0621 06:52:10.172922 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:10.172928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:10.173154 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:10.173319 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:10.174560 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:10.175553 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:10.176751 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:10.177320 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:10.177458 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:10.181221 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:10.181231 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:10.181246 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:10.181276 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:10.184159 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:10.184747 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:10.184754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:10.184756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:10.184865 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:10.185181 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:10.185184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:10.185186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:10.185504 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:10.185723 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:10.185892 (+ 169us) server_negotiation.cc:299] Negotiation successful
0621 06:52:10.186144 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":332,"threads_started":1}
W20260621 06:52:10.187685 24218 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59074
W20260621 06:52:10.188781 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 24095: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:52:10.188961 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 24095
I20260621 06:52:10.204548 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 24229
Jun 21 06:52:10 dist-test-slave-gmf2 krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024730, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:52:10.219441 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:10.205387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.20.120.194:43711)
0621 06:52:10.205994 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:10.206001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:10.206026 (+ 25us) server_negotiation.cc:407] Connection header received
0621 06:52:10.206085 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:10.206091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:10.206301 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:10.206433 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:10.207542 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:10.208516 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:10.209678 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:10.210337 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:10.210432 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:10.214478 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:10.214488 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:10.214501 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:10.214532 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:10.217076 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:10.217641 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:10.217647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:10.217650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:10.217761 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:10.218132 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:10.218135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:10.218137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:10.218433 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:10.218661 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:10.218857 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 06:52:10.219042 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":296,"threads_started":1}
W20260621 06:52:10.220741 24352 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44514
W20260621 06:52:10.221796 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 24229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:52:10.221963 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 24229
I20260621 06:52:10.237071 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 24362
Jun 21 06:52:10 dist-test-slave-gmf2 krb5kdc[23989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024730, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:52:10.253485 24503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:10.238071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.20.120.195:33947)
0621 06:52:10.238666 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:10.238673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:10.238700 (+ 27us) server_negotiation.cc:407] Connection header received
0621 06:52:10.238759 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:10.238765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:10.238974 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:10.239148 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:10.240461 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:10.241479 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:10.242803 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:10.243398 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:10.243536 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:10.248001 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:10.248013 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:10.248029 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:10.248064 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:10.251132 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:10.251762 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:10.251769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:10.251771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:10.251882 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:10.252209 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:10.252212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:10.252214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:10.252523 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:10.252736 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:10.252994 (+ 258us) server_negotiation.cc:299] Negotiation successful
0621 06:52:10.253199 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":313,"threads_started":1}
W20260621 06:52:10.254812 24485 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40852
W20260621 06:52:10.255975 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 24362: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:52:10.256155 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 24362
I20260621 06:52:10.271505 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 24002
I20260621 06:52:10.272686 24064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:10.375607 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 24002
2026-06-21T06:52:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3537 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:52:10 dist-test-slave-gmf2 krb5kdc[24507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:52:10 dist-test-slave-gmf2 krb5kdc[24507](info): set up 2 sockets
Jun 21 06:52:10 dist-test-slave-gmf2 krb5kdc[24507](info): commencing operation
krb5kdc: starting...
W20260621 06:52:11.494625 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:52:11 dist-test-slave-gmf2 krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:52:11.674852 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:42665
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:42665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:11.996455 24520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:11.996827 24520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:11.997004 24520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:12.005012 24520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:12.005154 24520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:12.005252 24520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:12.005354 24520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:12.005450 24520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:12.015080 24520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:42665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:42665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:12.016350 24520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:12.017663 24520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:12.029449 24525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:12.029369 24526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:12.029403 24528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:12.030362 24520 server_base.cc:1034] running on GCE node
I20260621 06:52:12.031198 24520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:12.032588 24520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:12.033821 24520 hybrid_clock.cc:648] HybridClock initialized: now 1782024732033780 us; error 54 us; skew 500 ppm
I20260621 06:52:12.034189 24520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:12 dist-test-slave-gmf2 krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024732, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:12.037389 24520 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:52:12.038842 24520 webserver.cc:460] Webserver started at http://127.20.120.254:39081/ using document root <none> and password file <none>
I20260621 06:52:12.039479 24520 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:12.039572 24520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:12.039824 24520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:12.042621 24520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "c08d8ae4dc574408937c30e819fde239"
format_stamp: "Formatted at 2026-06-21 06:52:12 on dist-test-slave-gmf2"
server_key: "dd9f8b1ef776b0fe8605cf8eb170230a"
server_key_iv: "8d0699cb28c4f342d7b138e8cb7392c0"
server_key_version: "encryptionkey@0"
I20260621 06:52:12.043349 24520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "c08d8ae4dc574408937c30e819fde239"
format_stamp: "Formatted at 2026-06-21 06:52:12 on dist-test-slave-gmf2"
server_key: "dd9f8b1ef776b0fe8605cf8eb170230a"
server_key_iv: "8d0699cb28c4f342d7b138e8cb7392c0"
server_key_version: "encryptionkey@0"
I20260621 06:52:12.048244 24520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:52:12.051818 24535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:12.053126 24520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:52:12.053272 24520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "c08d8ae4dc574408937c30e819fde239"
format_stamp: "Formatted at 2026-06-21 06:52:12 on dist-test-slave-gmf2"
server_key: "dd9f8b1ef776b0fe8605cf8eb170230a"
server_key_iv: "8d0699cb28c4f342d7b138e8cb7392c0"
server_key_version: "encryptionkey@0"
I20260621 06:52:12.053426 24520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:12.070631 24520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:12.074268 24520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:12.074481 24520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:12.092234 24520 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:42665
I20260621 06:52:12.092257 24586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:42665 every 8 connection(s)
I20260621 06:52:12.093609 24520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:12.097321 24587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:12.101779 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 24520
I20260621 06:52:12.102021 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:12.102797 20963 external_mini_cluster.cc:1428] Setting key f7b5a134dd5c9ad4ac2fe5a49b5a0920
I20260621 06:52:12.108546 24587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239: Bootstrap starting.
I20260621 06:52:12.111588 24587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:12.112576 24587 log.cc:824] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:12.116016 24587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239: No bootstrap required, opened a new log
Jun 21 06:52:12 dist-test-slave-gmf2 krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:12.126505 24590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:12.104813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57682 (local address 127.20.120.254:42665)
0621 06:52:12.105613 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:12.105628 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:12.105667 (+ 39us) server_negotiation.cc:407] Connection header received
0621 06:52:12.107462 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:12.107485 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:12.107884 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:12.108271 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:12.109783 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:12.111115 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:12.112463 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:12.113400 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:12.113610 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:12.118035 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:12.118055 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:12.118076 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:12.118137 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:12.122055 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:12.122701 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:12.122710 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:12.122718 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:12.122916 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:12.123261 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:12.123267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:12.123271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:12.123987 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:12.124297 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:12.124628 (+ 331us) server_negotiation.cc:299] Negotiation successful
0621 06:52:12.125089 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":386,"threads_started":1}
I20260621 06:52:12.126868 24587 raft_consensus.cc:348] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08d8ae4dc574408937c30e819fde239" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42665 } }
I20260621 06:52:12.127255 24587 raft_consensus.cc:374] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:12.127372 24587 raft_consensus.cc:729] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08d8ae4dc574408937c30e819fde239, State: Initialized, Role: FOLLOWER
I20260621 06:52:12.127923 24587 consensus_queue.cc:260] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08d8ae4dc574408937c30e819fde239" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42665 } }
I20260621 06:52:12.128189 24587 raft_consensus.cc:388] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:12.128329 24587 raft_consensus.cc:482] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:12.128520 24587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:12.130797 24587 raft_consensus.cc:504] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08d8ae4dc574408937c30e819fde239" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42665 } }
I20260621 06:52:12.131388 24587 leader_election.cc:302] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c08d8ae4dc574408937c30e819fde239; no voters:
I20260621 06:52:12.131969 24587 leader_election.cc:288] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:12.132100 24592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:12.132777 24592 raft_consensus.cc:686] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [term 1 LEADER]: Becoming Leader. State: Replica: c08d8ae4dc574408937c30e819fde239, State: Running, Role: LEADER
I20260621 06:52:12.133247 24592 consensus_queue.cc:237] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08d8ae4dc574408937c30e819fde239" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42665 } }
I20260621 06:52:12.133960 24587 sys_catalog.cc:564] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:12.136485 24594 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c08d8ae4dc574408937c30e819fde239. Latest consensus state: current_term: 1 leader_uuid: "c08d8ae4dc574408937c30e819fde239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08d8ae4dc574408937c30e819fde239" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42665 } } }
I20260621 06:52:12.136453 24593 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c08d8ae4dc574408937c30e819fde239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08d8ae4dc574408937c30e819fde239" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42665 } } }
I20260621 06:52:12.136902 24594 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:12.136902 24593 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:12.138569 24598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:12.146268 24598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:12.155958 24598 catalog_manager.cc:1347] Generated new cluster ID: b91452bbd8c74f749020806ad2b462e1
I20260621 06:52:12.156068 24598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:12.185191 24598 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:12.187067 24598 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:12.193495 24598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c08d8ae4dc574408937c30e819fde239: Generated new TSK 0
I20260621 06:52:12.194382 24598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:52:12.247510 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:42665
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:12.592162 24613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:12.592484 24613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:12.592582 24613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:12.600270 24613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:12.600404 24613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:12.600523 24613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:52:12.610904 24613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:42665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:12.612241 24613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:12.613549 24613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:12.627944 24618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:12.628026 24621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:12.627980 24619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:12.628908 24613 server_base.cc:1034] running on GCE node
I20260621 06:52:12.629634 24613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:12.630723 24613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:12.631963 24613 hybrid_clock.cc:648] HybridClock initialized: now 1782024732631910 us; error 50 us; skew 500 ppm
I20260621 06:52:12.632367 24613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:12 dist-test-slave-gmf2 krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024732, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:12.635823 24613 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:52:12.637429 24613 webserver.cc:460] Webserver started at http://127.20.120.193:39399/ using document root <none> and password file <none>
I20260621 06:52:12.638353 24613 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:12.638459 24613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:12.638720 24613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:12.641665 24613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "f75835215c3040e08106fa34e1ab8291"
format_stamp: "Formatted at 2026-06-21 06:52:12 on dist-test-slave-gmf2"
server_key: "1f231a3f8ea23c0e671ac32a391c2576"
server_key_iv: "0b6b35898f9969ad6df0cf45b3d8f50b"
server_key_version: "encryptionkey@0"
I20260621 06:52:12.642616 24613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "f75835215c3040e08106fa34e1ab8291"
format_stamp: "Formatted at 2026-06-21 06:52:12 on dist-test-slave-gmf2"
server_key: "1f231a3f8ea23c0e671ac32a391c2576"
server_key_iv: "0b6b35898f9969ad6df0cf45b3d8f50b"
server_key_version: "encryptionkey@0"
I20260621 06:52:12.648088 24613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:52:12.651774 24628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:12.653270 24613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 06:52:12.653461 24613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "f75835215c3040e08106fa34e1ab8291"
format_stamp: "Formatted at 2026-06-21 06:52:12 on dist-test-slave-gmf2"
server_key: "1f231a3f8ea23c0e671ac32a391c2576"
server_key_iv: "0b6b35898f9969ad6df0cf45b3d8f50b"
server_key_version: "encryptionkey@0"
I20260621 06:52:12.653672 24613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:12.688632 24613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:12.693252 24613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:12.693584 24613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:12.694698 24613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:12.696177 24613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:12.696261 24613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:12.696441 24613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:12.696507 24613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:12.730654 24613 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:44723
I20260621 06:52:12.730737 24740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:44723 every 8 connection(s)
I20260621 06:52:12.732066 24613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:52:12.737800 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 24613
I20260621 06:52:12.738065 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:52:12.738804 20963 external_mini_cluster.cc:1428] Setting key 35093015a48816244d30e90013360f5c
Jun 21 06:52:12 dist-test-slave-gmf2 krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024732, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:12.755728 24743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:12.734792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:60799 (local address 127.20.120.254:42665)
0621 06:52:12.735211 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:12.735216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:12.736313 (+ 1097us) server_negotiation.cc:407] Connection header received
0621 06:52:12.738118 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:12.738123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:12.738223 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:12.738355 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:12.740063 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:12.741049 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:12.743630 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:12.744058 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:12.744166 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:12.748630 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:12.748647 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:12.748655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:12.748701 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:12.752241 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:12.753020 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:12.753029 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:12.753036 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:12.753186 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:12.753610 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:12.753617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:12.753624 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:12.754053 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:12.754199 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:12.754965 (+ 766us) server_negotiation.cc:299] Negotiation successful
0621 06:52:12.755238 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":209,"threads_started":1}
I20260621 06:52:12.756394 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:12.735476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:42665 (local address 127.20.120.193:60799)
0621 06:52:12.736142 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:12.736180 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:12.737779 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:12.738573 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:12.738581 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:12.739180 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:12.739900 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:12.739921 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:12.741217 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:12.741231 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:12.743495 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:12.743513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:12.744310 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:12.744319 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:12.744653 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:12.745408 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:12.745448 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:12.748392 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:12.752403 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:12.752411 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:12.752426 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:12.752875 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:12.753306 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:12.753309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:12.753312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:12.753494 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:12.754180 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:12.754187 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:12.754546 (+ 359us) client_negotiation.cc:769] Sending connection context
0621 06:52:12.754787 (+ 241us) client_negotiation.cc:240] Negotiation successful
0621 06:52:12.755207 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":265,"threads_started":1}
I20260621 06:52:12.758277 24741 heartbeater.cc:347] Connected to a master server at 127.20.120.254:42665
I20260621 06:52:12.758610 24741 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:12.759467 24741 heartbeater.cc:511] Master 127.20.120.254:42665 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
I20260621 06:52:12.761693 24552 ts_manager.cc:194] Registered new tserver with Master: f75835215c3040e08106fa34e1ab8291 (127.20.120.193:44723)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
I20260621 06:52:12.763402 24552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:60799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:52:12.785467 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:42665
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:13.147822 24747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:13.148141 24747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:13.148296 24747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:13.155804 24747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:13.155926 24747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:13.156101 24747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:52:13.166946 24747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:42665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:13.168346 24747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:13.169687 24747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:13.182852 24752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:13.182888 24755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:13.182802 24753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:13.183838 24747 server_base.cc:1034] running on GCE node
I20260621 06:52:13.184530 24747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:13.185492 24747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:13.186721 24747 hybrid_clock.cc:648] HybridClock initialized: now 1782024733186666 us; error 46 us; skew 500 ppm
I20260621 06:52:13.187058 24747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:13 dist-test-slave-gmf2 krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024733, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:13.190200 24747 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:52:13.191742 24747 webserver.cc:460] Webserver started at http://127.20.120.194:43667/ using document root <none> and password file <none>
I20260621 06:52:13.192466 24747 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:13.192605 24747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:13.192919 24747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:13.195873 24747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "b835b52a47b140e8a440c830004dc56f"
format_stamp: "Formatted at 2026-06-21 06:52:13 on dist-test-slave-gmf2"
server_key: "f7ecd354b3bcddd757f8be6ccd9e35ce"
server_key_iv: "c44d7a33b47edf3171e3219f21cff106"
server_key_version: "encryptionkey@0"
I20260621 06:52:13.196658 24747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "b835b52a47b140e8a440c830004dc56f"
format_stamp: "Formatted at 2026-06-21 06:52:13 on dist-test-slave-gmf2"
server_key: "f7ecd354b3bcddd757f8be6ccd9e35ce"
server_key_iv: "c44d7a33b47edf3171e3219f21cff106"
server_key_version: "encryptionkey@0"
I20260621 06:52:13.201869 24747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 06:52:13.205401 24762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:13.206976 24747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:13.207101 24747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "b835b52a47b140e8a440c830004dc56f"
format_stamp: "Formatted at 2026-06-21 06:52:13 on dist-test-slave-gmf2"
server_key: "f7ecd354b3bcddd757f8be6ccd9e35ce"
server_key_iv: "c44d7a33b47edf3171e3219f21cff106"
server_key_version: "encryptionkey@0"
I20260621 06:52:13.207279 24747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:13.238847 24747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:13.243351 24747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:13.243602 24747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:13.244504 24747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:13.245819 24747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:13.245896 24747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:13.246004 24747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:13.246054 24747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:13.279929 24747 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:40233
I20260621 06:52:13.279965 24874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:40233 every 8 connection(s)
I20260621 06:52:13.281355 24747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:52:13.286542 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 24747
I20260621 06:52:13.286753 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:52:13.287205 20963 external_mini_cluster.cc:1428] Setting key ddc6f97e9996f7fd7dd29446e7b41fe4
Jun 21 06:52:13 dist-test-slave-gmf2 krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024733, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:13.301602 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:13.284084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:39555 (local address 127.20.120.254:42665)
0621 06:52:13.284551 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:13.284556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:13.285594 (+ 1038us) server_negotiation.cc:407] Connection header received
0621 06:52:13.287595 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:13.287600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:13.287692 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:13.287809 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:13.289532 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.290325 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.291872 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.292301 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.292430 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:13.295892 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:13.295897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:13.295900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:13.295930 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:13.298628 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:13.299343 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:13.299350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:13.299353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:13.299480 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:13.299913 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:13.299920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:13.299923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:13.300187 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:13.300348 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:13.301008 (+ 660us) server_negotiation.cc:299] Negotiation successful
0621 06:52:13.301221 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":256,"threads_started":1}
I20260621 06:52:13.302706 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:13.284718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:42665 (local address 127.20.120.194:39555)
0621 06:52:13.285422 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:13.285466 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:13.287314 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:13.288045 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:13.288053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:13.288756 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:13.289362 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:13.289380 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.290478 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.290483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:13.291712 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:13.291723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.292527 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.292533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:13.292776 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:13.293482 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:13.293504 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:13.295668 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:13.298781 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:13.298790 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:13.298807 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:13.299224 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:13.299602 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:13.299610 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:13.299614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:13.299803 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:13.300320 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:13.300329 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:13.300674 (+ 345us) client_negotiation.cc:769] Sending connection context
0621 06:52:13.300937 (+ 263us) client_negotiation.cc:240] Negotiation successful
0621 06:52:13.301473 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":302,"threads_started":1}
I20260621 06:52:13.304224 24875 heartbeater.cc:347] Connected to a master server at 127.20.120.254:42665
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
I20260621 06:52:13.304551 24875 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:13.305287 24875 heartbeater.cc:511] Master 127.20.120.254:42665 requested a full tablet report, sending...
I20260621 06:52:13.306962 24552 ts_manager.cc:194] Registered new tserver with Master: b835b52a47b140e8a440c830004dc56f (127.20.120.194:40233)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
I20260621 06:52:13.307992 24552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:39555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:52:13.331022 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:42665
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:13.656145 24881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:13.656469 24881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:13.656643 24881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:13.663833 24881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:13.663992 24881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:13.664166 24881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:52:13.673484 24881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:42665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:13.674810 24881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:13.676100 24881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:13.688922 24889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:13.689031 24886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:13.689002 24887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:13.690346 24881 server_base.cc:1034] running on GCE node
I20260621 06:52:13.691001 24881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:13.691960 24881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:13.693192 24881 hybrid_clock.cc:648] HybridClock initialized: now 1782024733693139 us; error 50 us; skew 500 ppm
I20260621 06:52:13.693497 24881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:13 dist-test-slave-gmf2 krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024733, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:13.696625 24881 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:52:13.698150 24881 webserver.cc:460] Webserver started at http://127.20.120.195:41099/ using document root <none> and password file <none>
I20260621 06:52:13.698765 24881 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:13.698858 24881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:13.699100 24881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:13.701803 24881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "ce96a20f5a8e4436b957f2c720093f5b"
format_stamp: "Formatted at 2026-06-21 06:52:13 on dist-test-slave-gmf2"
server_key: "7a2bb2625ec8f46013243cda26f33102"
server_key_iv: "71f99afa652ec52191a074a356cde6ce"
server_key_version: "encryptionkey@0"
I20260621 06:52:13.702697 24881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "ce96a20f5a8e4436b957f2c720093f5b"
format_stamp: "Formatted at 2026-06-21 06:52:13 on dist-test-slave-gmf2"
server_key: "7a2bb2625ec8f46013243cda26f33102"
server_key_iv: "71f99afa652ec52191a074a356cde6ce"
server_key_version: "encryptionkey@0"
I20260621 06:52:13.707535 24881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 06:52:13.710886 24896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:13.712179 24881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:52:13.712301 24881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "ce96a20f5a8e4436b957f2c720093f5b"
format_stamp: "Formatted at 2026-06-21 06:52:13 on dist-test-slave-gmf2"
server_key: "7a2bb2625ec8f46013243cda26f33102"
server_key_iv: "71f99afa652ec52191a074a356cde6ce"
server_key_version: "encryptionkey@0"
I20260621 06:52:13.712452 24881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:13.728359 24881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:13.732518 24881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:13.732798 24881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:13.733740 24881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:13.735332 24881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:13.735427 24881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:13.735572 24881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:13.735647 24881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:13.766427 24741 heartbeater.cc:503] Master 127.20.120.254:42665 was elected leader, sending a full tablet report...
I20260621 06:52:13.768770 24881 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:38911
I20260621 06:52:13.768748 25008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:38911 every 8 connection(s)
I20260621 06:52:13.770035 24881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:52:13.779098 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 24881
I20260621 06:52:13.779320 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:52:13.779824 20963 external_mini_cluster.cc:1428] Setting key 5001984874e2de4a390e16f00cd91b28
Jun 21 06:52:13 dist-test-slave-gmf2 krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024733, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:13.789252 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:13.772587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:49831 (local address 127.20.120.254:42665)
0621 06:52:13.772765 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:13.772770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:13.773824 (+ 1054us) server_negotiation.cc:407] Connection header received
0621 06:52:13.775648 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:13.775654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:13.775737 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:13.775857 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:13.777309 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.778300 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.780433 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.780875 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.781005 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:13.784224 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:13.784231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:13.784236 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:13.784277 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:13.786549 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:13.787210 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:13.787218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:13.787222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:13.787343 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:13.787748 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:13.787753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:13.787756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:13.788009 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:13.788164 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:13.788771 (+ 607us) server_negotiation.cc:299] Negotiation successful
0621 06:52:13.788955 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 06:52:13.790257 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:13.773002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:42665 (local address 127.20.120.195:49831)
0621 06:52:13.773676 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:13.773717 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:13.775381 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:13.776079 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:13.776088 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:13.776585 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:13.777150 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:13.777167 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.778451 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.778455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:13.780294 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:13.780309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.781081 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.781087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:13.781309 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:13.782010 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:13.782030 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:13.784030 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:13.786696 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:13.786705 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:13.786719 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:13.787088 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:13.787450 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:13.787456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:13.787464 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:13.787634 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:13.788130 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:13.788138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:13.788440 (+ 302us) client_negotiation.cc:769] Sending connection context
0621 06:52:13.788691 (+ 251us) client_negotiation.cc:240] Negotiation successful
0621 06:52:13.789125 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":250,"threads_started":1}
I20260621 06:52:13.791469 25009 heartbeater.cc:347] Connected to a master server at 127.20.120.254:42665
I20260621 06:52:13.791706 25009 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:13.792375 25009 heartbeater.cc:511] Master 127.20.120.254:42665 requested a full tablet report, sending...
I20260621 06:52:13.793864 24551 ts_manager.cc:194] Registered new tserver with Master: ce96a20f5a8e4436b957f2c720093f5b (127.20.120.195:38911)
I20260621 06:52:13.794750 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:52:13.794776 24551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:49831
Jun 21 06:52:13 dist-test-slave-gmf2 krb5kdc[24507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 06:52:13 dist-test-slave-gmf2 krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:52:13.842096 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:13.827283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.20.120.193:44723)
0621 06:52:13.827888 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:13.827896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:13.827923 (+ 27us) server_negotiation.cc:407] Connection header received
0621 06:52:13.827986 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:13.827992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:13.828225 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:13.828358 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:13.829720 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.830777 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.831938 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.832487 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.832578 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:13.836482 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:13.836492 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:13.836506 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:13.836536 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:13.839407 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:13.840080 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:13.840087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:13.840089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:13.840210 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:13.840568 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:13.840575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:13.840578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:13.840891 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:13.841213 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:13.841441 (+ 228us) server_negotiation.cc:299] Negotiation successful
0621 06:52:13.841689 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":316,"threads_started":1}
Jun 21 06:52:13 dist-test-slave-gmf2 krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:13.858922 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:13.846192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.20.120.254:42665)
0621 06:52:13.846428 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:13.846434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:13.846586 (+ 152us) server_negotiation.cc:407] Connection header received
0621 06:52:13.846734 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:13.846738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:13.846799 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:13.846879 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:13.848000 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.848728 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.849880 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.850339 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.850430 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:13.853989 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:13.853995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:13.853998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:13.854028 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:13.856370 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:13.856984 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:13.856989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:13.856992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:13.857194 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:13.857538 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:13.857541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:13.857543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:13.857863 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:13.858237 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:13.858410 (+ 173us) server_negotiation.cc:299] Negotiation successful
0621 06:52:13.858559 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260621 06:52:13.859437 24551 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57698
I20260621 06:52:13.860404 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 24613
I20260621 06:52:13.873035 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:13.861343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.20.120.193:44723)
0621 06:52:13.861514 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:13.861518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:13.861696 (+ 178us) server_negotiation.cc:407] Connection header received
0621 06:52:13.861816 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:13.861820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:13.861964 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:13.862085 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:13.863547 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.864196 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.865395 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.865816 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.865980 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:13.868155 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:13.868161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:13.868166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:13.868205 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:13.870657 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:13.871247 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:13.871252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:13.871256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:13.871375 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:13.871722 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:13.871727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:13.871730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:13.871978 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:13.872213 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:13.872508 (+ 295us) server_negotiation.cc:299] Negotiation successful
0621 06:52:13.872710 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 06:52:13.873680 24736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:13.954958 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 24613
I20260621 06:52:13.972488 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 24747
Jun 21 06:52:13 dist-test-slave-gmf2 krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:52:13.987418 25021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:13.973355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.20.120.194:40233)
0621 06:52:13.973870 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:13.973877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:13.973934 (+ 57us) server_negotiation.cc:407] Connection header received
0621 06:52:13.974005 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:13.974019 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:13.974244 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:13.974372 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:13.975406 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.976371 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.977596 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:13.978240 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:13.978332 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:13.982271 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:13.982282 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:13.982301 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:13.982331 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:13.985114 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:13.985685 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:13.985692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:13.985694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:13.985819 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:13.986180 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:13.986183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:13.986185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:13.986506 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:13.986724 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:13.986938 (+ 214us) server_negotiation.cc:299] Negotiation successful
0621 06:52:13.987120 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":264,"threads_started":1}
I20260621 06:52:13.988606 24870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:14.068977 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 24747
I20260621 06:52:14.084503 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 24881
Jun 21 06:52:14 dist-test-slave-gmf2 krb5kdc[24507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:52:14.101536 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:14.085431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.20.120.195:38911)
0621 06:52:14.086095 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:14.086103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:14.086154 (+ 51us) server_negotiation.cc:407] Connection header received
0621 06:52:14.086226 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:14.086246 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:14.086477 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:14.086601 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:14.087798 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:14.088775 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:14.090168 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:14.090693 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:14.090790 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:14.095899 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:14.095913 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:14.095929 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:14.095968 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:14.098850 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:14.099471 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:14.099489 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:14.099493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:14.099617 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:14.100008 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:14.100019 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:14.100022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:14.100359 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:14.100665 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:14.100957 (+ 292us) server_negotiation.cc:299] Negotiation successful
0621 06:52:14.101209 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":277,"threads_started":1}
I20260621 06:52:14.102815 25004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:14.183962 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 24881
I20260621 06:52:14.200364 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 24520
I20260621 06:52:14.201550 24582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:14.302366 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 24520
2026-06-21T06:52:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3926 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:52:14 dist-test-slave-gmf2 krb5kdc[25028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:52:14 dist-test-slave-gmf2 krb5kdc[25028](info): set up 2 sockets
Jun 21 06:52:14 dist-test-slave-gmf2 krb5kdc[25028](info): commencing operation
krb5kdc: starting...
W20260621 06:52:15.378371 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:52:15 dist-test-slave-gmf2 krb5kdc[25028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:52:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:52:15.578142 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34549
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:15.891305 25041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:15.891589 25041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:15.891772 25041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:15.898896 25041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:15.899032 25041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:15.899119 25041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:15.899228 25041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:15.899312 25041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:15.908965 25041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:34549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:15.910341 25041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:15.911629 25041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:15.922946 25046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:15.922842 25047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:15.922911 25049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:15.924507 25041 server_base.cc:1034] running on GCE node
I20260621 06:52:15.925247 25041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:15.926651 25041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:15.927913 25041 hybrid_clock.cc:648] HybridClock initialized: now 1782024735927882 us; error 65 us; skew 500 ppm
I20260621 06:52:15.928249 25041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:15 dist-test-slave-gmf2 krb5kdc[25028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024735, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:15.934429 25041 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:52:15.935676 25041 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:15.935817 25041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:15.936169 25041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:15.939347 25041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "69f0e4eb82474914b2e0939235322f59"
format_stamp: "Formatted at 2026-06-21 06:52:15 on dist-test-slave-gmf2"
server_key: "03c570f3b5ad005bdc00673d86e9696e"
server_key_iv: "bb517dadadccd9c7b7032c4490f0b148"
server_key_version: "encryptionkey@0"
I20260621 06:52:15.940244 25041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "69f0e4eb82474914b2e0939235322f59"
format_stamp: "Formatted at 2026-06-21 06:52:15 on dist-test-slave-gmf2"
server_key: "03c570f3b5ad005bdc00673d86e9696e"
server_key_iv: "bb517dadadccd9c7b7032c4490f0b148"
server_key_version: "encryptionkey@0"
I20260621 06:52:15.945331 25041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 06:52:15.948868 25055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:15.950214 25041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:15.950410 25041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "69f0e4eb82474914b2e0939235322f59"
format_stamp: "Formatted at 2026-06-21 06:52:15 on dist-test-slave-gmf2"
server_key: "03c570f3b5ad005bdc00673d86e9696e"
server_key_iv: "bb517dadadccd9c7b7032c4490f0b148"
server_key_version: "encryptionkey@0"
I20260621 06:52:15.950667 25041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:15.964524 25041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:15.969202 25041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:15.969506 25041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:15.989647 25041 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:34549
I20260621 06:52:15.989624 25106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:34549 every 8 connection(s)
I20260621 06:52:15.990674 25041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:15.994766 25107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:15.996002 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 25041
I20260621 06:52:15.996235 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:15.997047 20963 external_mini_cluster.cc:1428] Setting key 29ef5ad99f872a71f62a4d17acc34344
I20260621 06:52:16.009119 25107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59: Bootstrap starting.
Jun 21 06:52:16 dist-test-slave-gmf2 krb5kdc[25028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:16.013168 25107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:16.014303 25107 log.cc:824] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:16.017429 25107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59: No bootstrap required, opened a new log
I20260621 06:52:16.020588 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:15.999186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.20.120.254:34549)
0621 06:52:15.999966 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:15.999980 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:16.000022 (+ 42us) server_negotiation.cc:407] Connection header received
0621 06:52:16.001480 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:16.001504 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:16.001881 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:16.002265 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:16.004110 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:16.005395 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:16.007245 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:16.007899 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:16.008033 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:16.012890 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:16.012898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:16.012913 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:16.012952 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:16.016583 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:16.017254 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:16.017261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:16.017269 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:16.017485 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:16.017867 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:16.017874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:16.017877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:16.018434 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:16.018749 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:16.019178 (+ 429us) server_negotiation.cc:299] Negotiation successful
0621 06:52:16.019592 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":395,"threads_started":1}
I20260621 06:52:16.025401 25107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f0e4eb82474914b2e0939235322f59" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34549 } }
I20260621 06:52:16.025781 25107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:16.025900 25107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f0e4eb82474914b2e0939235322f59, State: Initialized, Role: FOLLOWER
I20260621 06:52:16.026470 25107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f0e4eb82474914b2e0939235322f59" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34549 } }
I20260621 06:52:16.026687 25107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:16.026808 25107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:16.026953 25107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:16.029305 25107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f0e4eb82474914b2e0939235322f59" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34549 } }
I20260621 06:52:16.029863 25107 leader_election.cc:302] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f0e4eb82474914b2e0939235322f59; no voters:
I20260621 06:52:16.030476 25107 leader_election.cc:288] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:16.030609 25112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:16.031257 25112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [term 1 LEADER]: Becoming Leader. State: Replica: 69f0e4eb82474914b2e0939235322f59, State: Running, Role: LEADER
I20260621 06:52:16.031788 25112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f0e4eb82474914b2e0939235322f59" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34549 } }
I20260621 06:52:16.032550 25107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:16.034700 25114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69f0e4eb82474914b2e0939235322f59. Latest consensus state: current_term: 1 leader_uuid: "69f0e4eb82474914b2e0939235322f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f0e4eb82474914b2e0939235322f59" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34549 } } }
I20260621 06:52:16.034668 25113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69f0e4eb82474914b2e0939235322f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f0e4eb82474914b2e0939235322f59" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34549 } } }
I20260621 06:52:16.035070 25114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:16.035142 25113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:16.041975 25119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:16.047883 25119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:16.056567 25119 catalog_manager.cc:1347] Generated new cluster ID: 4b1ad764dfe741e8a469e8657229faa3
I20260621 06:52:16.056702 25119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:16.074556 25119 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:16.076220 25119 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:16.084535 25119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59: Generated new TSK 0
I20260621 06:52:16.085317 25119 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:52:16.135951 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34549
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:16.451331 25133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:16.451637 25133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:16.451736 25133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:16.459345 25133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:16.459453 25133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:16.459574 25133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:52:16.469095 25133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:16.470433 25133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:16.471668 25133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:16.484457 25138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:16.484423 25141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:16.484505 25139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:16.485464 25133 server_base.cc:1034] running on GCE node
I20260621 06:52:16.486171 25133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:16.487129 25133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:16.488333 25133 hybrid_clock.cc:648] HybridClock initialized: now 1782024736488291 us; error 33 us; skew 500 ppm
I20260621 06:52:16.488616 25133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:16 dist-test-slave-gmf2 krb5kdc[25028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024736, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:16.491740 25133 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:52:16.492854 25133 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:16.492939 25133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:16.493221 25133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:16.496112 25133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "0014bb33a7d645a29402c1cc179fdc7d"
format_stamp: "Formatted at 2026-06-21 06:52:16 on dist-test-slave-gmf2"
server_key: "834cea9a16fc8a483a1059ccebf0ab80"
server_key_iv: "f6e86a35cb66ada5b2fbd66f8301f87d"
server_key_version: "encryptionkey@0"
I20260621 06:52:16.496922 25133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "0014bb33a7d645a29402c1cc179fdc7d"
format_stamp: "Formatted at 2026-06-21 06:52:16 on dist-test-slave-gmf2"
server_key: "834cea9a16fc8a483a1059ccebf0ab80"
server_key_iv: "f6e86a35cb66ada5b2fbd66f8301f87d"
server_key_version: "encryptionkey@0"
I20260621 06:52:16.501820 25133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260621 06:52:16.505162 25147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:16.506616 25133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:16.506783 25133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "0014bb33a7d645a29402c1cc179fdc7d"
format_stamp: "Formatted at 2026-06-21 06:52:16 on dist-test-slave-gmf2"
server_key: "834cea9a16fc8a483a1059ccebf0ab80"
server_key_iv: "f6e86a35cb66ada5b2fbd66f8301f87d"
server_key_version: "encryptionkey@0"
I20260621 06:52:16.506995 25133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:16.552721 25133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:16.557366 25133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:16.557649 25133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:16.558653 25133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:16.560001 25133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:16.560086 25133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:16.560240 25133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:16.560334 25133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:16.592965 25133 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:35247
I20260621 06:52:16.593050 25259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:35247 every 8 connection(s)
I20260621 06:52:16.593851 25133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:52:16.594686 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 25133
I20260621 06:52:16.594901 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:52:16.595579 20963 external_mini_cluster.cc:1428] Setting key a966c0b03cd6a062103a73e6c1da81aa
Jun 21 06:52:16 dist-test-slave-gmf2 krb5kdc[25028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024736, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
I20260621 06:52:16.616184 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:16.596503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:36577 (local address 127.20.120.254:34549)
0621 06:52:16.596990 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:16.596996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:16.598167 (+ 1171us) server_negotiation.cc:407] Connection header received
0621 06:52:16.599914 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:16.599919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:16.599998 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:16.600119 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:16.601638 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:16.602544 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:16.604695 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:16.605078 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:16.605175 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:16.609315 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:16.609325 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:16.609333 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:16.609378 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:16.613111 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:16.613822 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:16.613829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:16.613834 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:16.614009 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:16.614404 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:16.614411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:16.614416 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:16.614788 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:16.614933 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:16.615612 (+ 679us) server_negotiation.cc:299] Negotiation successful
0621 06:52:16.615817 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":265,"threads_started":1}
I20260621 06:52:16.617300 25263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:16.597139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34549 (local address 127.20.120.193:36577)
0621 06:52:16.598033 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:16.598071 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:16.599663 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:16.600368 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:16.600376 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:16.600899 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:16.601468 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:16.601487 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:16.602714 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:16.602718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:16.604529 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:16.604546 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:16.605312 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:16.605321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:16.605632 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:16.606354 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:16.606411 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:16.609083 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:16.613253 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:16.613261 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:16.613274 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:16.613700 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:16.614117 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:16.614120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:16.614123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:16.614293 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:16.614913 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:16.614919 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:16.615242 (+ 323us) client_negotiation.cc:769] Sending connection context
0621 06:52:16.615508 (+ 266us) client_negotiation.cc:240] Negotiation successful
0621 06:52:16.616057 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":361,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
I20260621 06:52:16.619099 25260 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34549
I20260621 06:52:16.619408 25260 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:16.620102 25260 heartbeater.cc:511] Master 127.20.120.254:34549 requested a full tablet report, sending...
I20260621 06:52:16.622509 25072 ts_manager.cc:194] Registered new tserver with Master: 0014bb33a7d645a29402c1cc179fdc7d (127.20.120.193:35247)
I20260621 06:52:16.624127 25072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:36577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:52:16.641287 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34549
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:16.962265 25266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:16.962569 25266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:16.962666 25266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:16.969693 25266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:16.969805 25266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:16.969933 25266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:52:16.980690 25266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:16.981943 25266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:16.983273 25266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:16.998136 25272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:16.998507 25271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:16.998111 25274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:16.999061 25266 server_base.cc:1034] running on GCE node
I20260621 06:52:16.999666 25266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:17.000620 25266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:17.001849 25266 hybrid_clock.cc:648] HybridClock initialized: now 1782024737001789 us; error 52 us; skew 500 ppm
I20260621 06:52:17.002195 25266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:16 dist-test-slave-gmf2 krb5kdc[25028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024736, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:17.005381 25266 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:52:17.006587 25266 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:17.006677 25266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:17.007022 25266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:17.009871 25266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "5cca488f84f34ba29f9601fe6c03363e"
format_stamp: "Formatted at 2026-06-21 06:52:17 on dist-test-slave-gmf2"
server_key: "13c0dee377d679d2a74969d2dd5edc87"
server_key_iv: "51e07600559dfc679fa72f5e18a9c5bd"
server_key_version: "encryptionkey@0"
I20260621 06:52:17.010718 25266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "5cca488f84f34ba29f9601fe6c03363e"
format_stamp: "Formatted at 2026-06-21 06:52:17 on dist-test-slave-gmf2"
server_key: "13c0dee377d679d2a74969d2dd5edc87"
server_key_iv: "51e07600559dfc679fa72f5e18a9c5bd"
server_key_version: "encryptionkey@0"
I20260621 06:52:17.015684 25266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 06:52:17.018970 25280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:17.020284 25266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:52:17.020406 25266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "5cca488f84f34ba29f9601fe6c03363e"
format_stamp: "Formatted at 2026-06-21 06:52:17 on dist-test-slave-gmf2"
server_key: "13c0dee377d679d2a74969d2dd5edc87"
server_key_iv: "51e07600559dfc679fa72f5e18a9c5bd"
server_key_version: "encryptionkey@0"
I20260621 06:52:17.020581 25266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:17.038668 25266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:17.042676 25266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:17.042951 25266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:17.043938 25266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:17.045145 25266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:17.045228 25266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:17.045310 25266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:17.045370 25266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:17.077005 25266 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:39635
I20260621 06:52:17.077052 25392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:39635 every 8 connection(s)
I20260621 06:52:17.077853 25266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:52:17.080741 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 25266
I20260621 06:52:17.080936 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:52:17.081497 20963 external_mini_cluster.cc:1428] Setting key 39eaf4c95dfc53f88d6343f8f774f6ad
Jun 21 06:52:17 dist-test-slave-gmf2 krb5kdc[25028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024736, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:17.099426 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.080324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:46519 (local address 127.20.120.254:34549)
0621 06:52:17.080540 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:17.080547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:17.081860 (+ 1313us) server_negotiation.cc:407] Connection header received
0621 06:52:17.084155 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:17.084158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:17.084231 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:17.084322 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:17.086402 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.087067 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.089593 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.090063 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.090165 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.093740 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:17.093748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:17.093774 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:17.093804 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:17.096413 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:17.097223 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:17.097229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:17.097233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:17.097353 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:17.097760 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:17.097766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:17.097769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:17.098081 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:17.098232 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:17.098886 (+ 654us) server_negotiation.cc:299] Negotiation successful
0621 06:52:17.099098 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 06:52:17.100517 25395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.080800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34549 (local address 127.20.120.194:46519)
0621 06:52:17.081650 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:17.081734 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:17.083854 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:17.084573 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:17.084587 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:17.085389 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:17.086235 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.086260 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.087222 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.087230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.089372 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.089388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.090295 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.090304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.090606 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.091363 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:17.091385 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:17.093519 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:17.096550 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:17.096558 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:17.096574 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:17.097103 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:17.097465 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:17.097471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:17.097475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:17.097644 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:17.098206 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:17.098213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:17.098530 (+ 317us) client_negotiation.cc:769] Sending connection context
0621 06:52:17.098798 (+ 268us) client_negotiation.cc:240] Negotiation successful
0621 06:52:17.099315 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":340,"threads_started":1}
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
I20260621 06:52:17.102566 25393 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34549
I20260621 06:52:17.102902 25393 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
I20260621 06:52:17.103689 25393 heartbeater.cc:511] Master 127.20.120.254:34549 requested a full tablet report, sending...
I20260621 06:52:17.105304 25072 ts_manager.cc:194] Registered new tserver with Master: 5cca488f84f34ba29f9601fe6c03363e (127.20.120.194:39635)
I20260621 06:52:17.106551 25072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:46519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:52:17.126751 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34549
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:17.438376 25398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:17.438668 25398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:17.438827 25398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:17.447505 25398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:17.447652 25398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:17.447822 25398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:52:17.457619 25398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:17.458923 25398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:17.460455 25398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:17.472987 25403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:17.473847 25404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:17.474210 25398 server_base.cc:1034] running on GCE node
W20260621 06:52:17.473925 25406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:17.474859 25398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:17.475754 25398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:17.476976 25398 hybrid_clock.cc:648] HybridClock initialized: now 1782024737476936 us; error 41 us; skew 500 ppm
I20260621 06:52:17.477290 25398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:17 dist-test-slave-gmf2 krb5kdc[25028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024737, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:17.480608 25398 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:52:17.481812 25398 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:17.481972 25398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:17.482291 25398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:17.485090 25398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "c85c8e1b739e4643921ab60273d16bbf"
format_stamp: "Formatted at 2026-06-21 06:52:17 on dist-test-slave-gmf2"
server_key: "09081b3aa2dbe2b073ab4f9f8b75da5a"
server_key_iv: "683e40dbb00a3dbc14d1d6e9858317a3"
server_key_version: "encryptionkey@0"
I20260621 06:52:17.485896 25398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "c85c8e1b739e4643921ab60273d16bbf"
format_stamp: "Formatted at 2026-06-21 06:52:17 on dist-test-slave-gmf2"
server_key: "09081b3aa2dbe2b073ab4f9f8b75da5a"
server_key_iv: "683e40dbb00a3dbc14d1d6e9858317a3"
server_key_version: "encryptionkey@0"
I20260621 06:52:17.490885 25398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 06:52:17.494249 25412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:17.495471 25398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 06:52:17.495639 25398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "c85c8e1b739e4643921ab60273d16bbf"
format_stamp: "Formatted at 2026-06-21 06:52:17 on dist-test-slave-gmf2"
server_key: "09081b3aa2dbe2b073ab4f9f8b75da5a"
server_key_iv: "683e40dbb00a3dbc14d1d6e9858317a3"
server_key_version: "encryptionkey@0"
I20260621 06:52:17.495877 25398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:17.513638 25398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:17.518085 25398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:17.518357 25398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:17.519277 25398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:17.520744 25398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:17.520860 25398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:17.520996 25398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:17.521083 25398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:17.552855 25398 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:37951
I20260621 06:52:17.552912 25524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:37951 every 8 connection(s)
I20260621 06:52:17.553727 25398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:52:17.554466 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 25398
I20260621 06:52:17.554661 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:52:17.555195 20963 external_mini_cluster.cc:1428] Setting key 2322311088f1c89a598165b5a15ff070
Jun 21 06:52:17 dist-test-slave-gmf2 krb5kdc[25028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024737, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:17.573720 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.556505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:32839 (local address 127.20.120.254:34549)
0621 06:52:17.556698 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:17.556704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:17.558200 (+ 1496us) server_negotiation.cc:407] Connection header received
0621 06:52:17.560072 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:17.560076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:17.560139 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:17.560218 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:17.561750 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.562540 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.563997 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.564390 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.564482 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.567766 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:17.567770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:17.567774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:17.567802 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:17.570847 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:17.571533 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:17.571540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:17.571544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:17.571679 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:17.572102 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:17.572107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:17.572110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:17.572365 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:17.572517 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:17.573156 (+ 639us) server_negotiation.cc:299] Negotiation successful
0621 06:52:17.573363 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 06:52:17.574693 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.557118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34549 (local address 127.20.120.195:32839)
0621 06:52:17.558022 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:17.558081 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:17.559832 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:17.560446 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:17.560454 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:17.560979 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:17.561592 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.561609 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.562665 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.562668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.563873 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.563883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.564562 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.564566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.564739 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.565432 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:17.565454 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:17.567589 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:17.570971 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:17.570979 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:17.570996 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:17.571416 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:17.571807 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:17.571812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:17.571816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:17.571983 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:17.572498 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:17.572505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:17.572824 (+ 319us) client_negotiation.cc:769] Sending connection context
0621 06:52:17.573081 (+ 257us) client_negotiation.cc:240] Negotiation successful
0621 06:52:17.573592 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":378,"threads_started":1}
I20260621 06:52:17.576068 25525 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34549
I20260621 06:52:17.576401 25525 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:17.577028 25525 heartbeater.cc:511] Master 127.20.120.254:34549 requested a full tablet report, sending...
I20260621 06:52:17.578361 25072 ts_manager.cc:194] Registered new tserver with Master: c85c8e1b739e4643921ab60273d16bbf (127.20.120.195:37951)
I20260621 06:52:17.579047 25072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:32839
I20260621 06:52:17.582880 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:52:17.599773 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.586985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.20.120.254:34549)
0621 06:52:17.587170 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:17.587174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:17.587559 (+ 385us) server_negotiation.cc:407] Connection header received
0621 06:52:17.587686 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:17.587690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:17.587750 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:17.587841 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:17.589193 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.589864 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.591203 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.591682 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.591781 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.594094 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:17.594100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:17.594104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:17.594140 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:17.597274 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:17.597886 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:17.597891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:17.597894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:17.598092 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:17.598430 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:17.598435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:17.598439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:17.598825 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:17.599086 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:17.599351 (+ 265us) server_negotiation.cc:299] Negotiation successful
0621 06:52:17.599561 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 06:52:17.603668 25072 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:52:17.610356 25072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:52:17.627204 25260 heartbeater.cc:503] Master 127.20.120.254:34549 was elected leader, sending a full tablet report...
I20260621 06:52:17.641863 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.622098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.20.120.193:35247)
0621 06:52:17.633849 (+ 11751us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:17.633860 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:17.633893 (+ 33us) server_negotiation.cc:407] Connection header received
0621 06:52:17.634106 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:17.634115 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:17.634516 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:17.634709 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:17.635418 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.636970 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.639420 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.640713 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.640909 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.641076 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:17.641370 (+ 294us) server_negotiation.cc:299] Negotiation successful
0621 06:52:17.641534 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11578,"thread_start_us":11441,"threads_started":1}
I20260621 06:52:17.641987 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.621763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:35247 (local address 127.0.0.1:58518)
0621 06:52:17.622986 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:17.623024 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:17.623168 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:17.634756 (+ 11588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:17.634766 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:17.634804 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:17.635197 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.635213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.637128 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.637133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.639238 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.639251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.640908 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.640914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.641097 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.641144 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 06:52:17.641309 (+ 165us) client_negotiation.cc:240] Negotiation successful
0621 06:52:17.641489 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":2,"thread_start_us":12076,"threads_started":1}
I20260621 06:52:17.642601 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.622292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:39635 (local address 127.0.0.1:56438)
0621 06:52:17.634138 (+ 11846us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:17.634161 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:17.634269 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:17.635241 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:17.635246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:17.635273 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:17.635816 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.635829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.637517 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.637523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.639699 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.639713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.641735 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.641741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.641956 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.641998 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 06:52:17.642100 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 06:52:17.642213 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11730,"mutex_wait_us":57,"thread_start_us":350,"threads_started":1}
I20260621 06:52:17.643411 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.622699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.20.120.194:39635)
0621 06:52:17.634425 (+ 11726us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:17.634432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:17.634457 (+ 25us) server_negotiation.cc:407] Connection header received
0621 06:52:17.634554 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:17.634562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:17.634803 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:17.634997 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:17.635986 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.637386 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.639864 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.641246 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.641431 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.641579 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:17.642917 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0621 06:52:17.643098 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10555,"thread_start_us":10400,"threads_started":1}
I20260621 06:52:17.643743 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.634533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.20.120.195:37951)
0621 06:52:17.635346 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:17.635356 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:17.636063 (+ 707us) server_negotiation.cc:407] Connection header received
0621 06:52:17.636241 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:17.636246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:17.636398 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:17.636571 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:17.637130 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.638608 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.640608 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.642119 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.642312 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.642483 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:17.643107 (+ 624us) server_negotiation.cc:299] Negotiation successful
0621 06:52:17.643278 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":485,"threads_started":1}
I20260621 06:52:17.644430 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.635328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:37951 (local address 127.0.0.1:44598)
0621 06:52:17.635951 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:17.635972 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:17.636096 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:17.636558 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:17.636562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:17.636583 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:17.636943 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.636951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.638772 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.638777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.640454 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.640464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.642632 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.642640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.642871 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.642903 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 06:52:17.643065 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 06:52:17.643188 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":1,"thread_start_us":329,"threads_started":1}
I20260621 06:52:17.646572 25195 tablet_service.cc:1467] Processing CreateTablet for tablet fdbbc16b8b974b1cb70a136d4864af7c (DEFAULT_TABLE table=test-table [id=e051841e5ad341f3a5d8912821eea110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:17.646981 25328 tablet_service.cc:1467] Processing CreateTablet for tablet fdbbc16b8b974b1cb70a136d4864af7c (DEFAULT_TABLE table=test-table [id=e051841e5ad341f3a5d8912821eea110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:17.647876 25195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbbc16b8b974b1cb70a136d4864af7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:17.648326 25328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbbc16b8b974b1cb70a136d4864af7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:17.648272 25460 tablet_service.cc:1467] Processing CreateTablet for tablet fdbbc16b8b974b1cb70a136d4864af7c (DEFAULT_TABLE table=test-table [id=e051841e5ad341f3a5d8912821eea110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:17.649511 25460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbbc16b8b974b1cb70a136d4864af7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:17.659118 25543 tablet_bootstrap.cc:492] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d: Bootstrap starting.
I20260621 06:52:17.662400 25544 tablet_bootstrap.cc:492] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e: Bootstrap starting.
I20260621 06:52:17.663578 25543 tablet_bootstrap.cc:654] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:17.664898 25543 log.cc:824] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:17.665968 25544 tablet_bootstrap.cc:654] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:17.667227 25544 log.cc:824] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:17.668220 25543 tablet_bootstrap.cc:492] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d: No bootstrap required, opened a new log
I20260621 06:52:17.668507 25543 ts_tablet_manager.cc:1397] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260621 06:52:17.670022 25544 tablet_bootstrap.cc:492] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e: No bootstrap required, opened a new log
I20260621 06:52:17.670251 25544 ts_tablet_manager.cc:1397] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260621 06:52:17.670598 25545 tablet_bootstrap.cc:492] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf: Bootstrap starting.
I20260621 06:52:17.673528 25545 tablet_bootstrap.cc:654] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:17.674538 25545 log.cc:824] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:17.676963 25545 tablet_bootstrap.cc:492] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf: No bootstrap required, opened a new log
I20260621 06:52:17.677212 25545 ts_tablet_manager.cc:1397] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260621 06:52:17.679050 25543 raft_consensus.cc:348] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0014bb33a7d645a29402c1cc179fdc7d" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35247 } } peers { permanent_uuid: "5cca488f84f34ba29f9601fe6c03363e" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 39635 } } peers { permanent_uuid: "c85c8e1b739e4643921ab60273d16bbf" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37951 } }
I20260621 06:52:17.679490 25543 raft_consensus.cc:374] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:17.679620 25543 raft_consensus.cc:729] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0014bb33a7d645a29402c1cc179fdc7d, State: Initialized, Role: FOLLOWER
I20260621 06:52:17.680227 25543 consensus_queue.cc:260] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0014bb33a7d645a29402c1cc179fdc7d" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35247 } } peers { permanent_uuid: "5cca488f84f34ba29f9601fe6c03363e" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 39635 } } peers { permanent_uuid: "c85c8e1b739e4643921ab60273d16bbf" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37951 } }
I20260621 06:52:17.680498 25544 raft_consensus.cc:348] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0014bb33a7d645a29402c1cc179fdc7d" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35247 } } peers { permanent_uuid: "5cca488f84f34ba29f9601fe6c03363e" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 39635 } } peers { permanent_uuid: "c85c8e1b739e4643921ab60273d16bbf" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37951 } }
I20260621 06:52:17.681015 25544 raft_consensus.cc:374] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:17.681147 25544 raft_consensus.cc:729] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cca488f84f34ba29f9601fe6c03363e, State: Initialized, Role: FOLLOWER
I20260621 06:52:17.681710 25544 consensus_queue.cc:260] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0014bb33a7d645a29402c1cc179fdc7d" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35247 } } peers { permanent_uuid: "5cca488f84f34ba29f9601fe6c03363e" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 39635 } } peers { permanent_uuid: "c85c8e1b739e4643921ab60273d16bbf" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37951 } }
I20260621 06:52:17.682080 25543 ts_tablet_manager.cc:1428] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260621 06:52:17.683095 25393 heartbeater.cc:503] Master 127.20.120.254:34549 was elected leader, sending a full tablet report...
I20260621 06:52:17.683661 25544 ts_tablet_manager.cc:1428] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260621 06:52:17.687464 25545 raft_consensus.cc:348] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0014bb33a7d645a29402c1cc179fdc7d" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35247 } } peers { permanent_uuid: "5cca488f84f34ba29f9601fe6c03363e" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 39635 } } peers { permanent_uuid: "c85c8e1b739e4643921ab60273d16bbf" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37951 } }
I20260621 06:52:17.687896 25545 raft_consensus.cc:374] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:17.688014 25545 raft_consensus.cc:729] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85c8e1b739e4643921ab60273d16bbf, State: Initialized, Role: FOLLOWER
I20260621 06:52:17.688583 25545 consensus_queue.cc:260] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0014bb33a7d645a29402c1cc179fdc7d" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35247 } } peers { permanent_uuid: "5cca488f84f34ba29f9601fe6c03363e" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 39635 } } peers { permanent_uuid: "c85c8e1b739e4643921ab60273d16bbf" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37951 } }
I20260621 06:52:17.689502 25525 heartbeater.cc:503] Master 127.20.120.254:34549 was elected leader, sending a full tablet report...
I20260621 06:52:17.690090 25545 ts_tablet_manager.cc:1428] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260621 06:52:17.798144 25551 raft_consensus.cc:482] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:52:17.798439 25551 raft_consensus.cc:504] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0014bb33a7d645a29402c1cc179fdc7d" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35247 } } peers { permanent_uuid: "5cca488f84f34ba29f9601fe6c03363e" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 39635 } } peers { permanent_uuid: "c85c8e1b739e4643921ab60273d16bbf" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37951 } }
I20260621 06:52:17.800087 25551 leader_election.cc:288] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0014bb33a7d645a29402c1cc179fdc7d (127.20.120.193:35247), 5cca488f84f34ba29f9601fe6c03363e (127.20.120.194:39635)
W20260621 06:52:17.805243 25526 tablet.cc:2369] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:52:17.806316 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.800677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:39983 (local address 127.20.120.194:39635)
0621 06:52:17.800962 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:17.800969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:17.800999 (+ 30us) server_negotiation.cc:407] Connection header received
0621 06:52:17.801082 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:17.801089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:17.801175 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:17.801290 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:17.801759 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.802699 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.804443 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.805436 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.805577 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.805650 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:17.805956 (+ 306us) server_negotiation.cc:299] Negotiation successful
0621 06:52:17.806075 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260621 06:52:17.806407 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.800394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:39635 (local address 127.20.120.195:39983)
0621 06:52:17.800740 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:17.800762 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:17.800928 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:17.801311 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:17.801314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:17.801365 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:17.801602 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.801614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.802831 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.802836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.804274 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.804284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.805578 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.805584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.805767 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.805791 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 06:52:17.805898 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 06:52:17.806046 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":3,"spinlock_wait_cycles":28544}
I20260621 06:52:17.807102 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.800511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:55859 (local address 127.20.120.193:35247)
0621 06:52:17.800732 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:17.800740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:17.801378 (+ 638us) server_negotiation.cc:407] Connection header received
0621 06:52:17.801552 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:17.801559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:17.801628 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:17.801717 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:17.802297 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.803135 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.804908 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.806110 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.806299 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.806368 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:17.806732 (+ 364us) server_negotiation.cc:299] Negotiation successful
0621 06:52:17.806871 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260621 06:52:17.807099 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.800513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:35247 (local address 127.20.120.195:55859)
0621 06:52:17.801248 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:17.801277 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:17.801400 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:17.801749 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:17.801753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:17.801776 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:17.802134 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.802146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.803294 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.803297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.804753 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:17.804766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.806325 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.806331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:17.806499 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.806529 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 06:52:17.806660 (+ 131us) client_negotiation.cc:240] Negotiation successful
0621 06:52:17.806798 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"spinlock_wait_cycles":1280,"thread_start_us":414,"threads_started":1}
I20260621 06:52:17.807561 25348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbbc16b8b974b1cb70a136d4864af7c" candidate_uuid: "c85c8e1b739e4643921ab60273d16bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cca488f84f34ba29f9601fe6c03363e" is_pre_election: true
I20260621 06:52:17.808158 25348 raft_consensus.cc:2454] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c85c8e1b739e4643921ab60273d16bbf in term 0.
I20260621 06:52:17.808115 25215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbbc16b8b974b1cb70a136d4864af7c" candidate_uuid: "c85c8e1b739e4643921ab60273d16bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0014bb33a7d645a29402c1cc179fdc7d" is_pre_election: true
I20260621 06:52:17.808656 25215 raft_consensus.cc:2454] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c85c8e1b739e4643921ab60273d16bbf in term 0.
I20260621 06:52:17.808883 25414 leader_election.cc:302] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cca488f84f34ba29f9601fe6c03363e, c85c8e1b739e4643921ab60273d16bbf; no voters:
I20260621 06:52:17.809299 25551 raft_consensus.cc:2781] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:52:17.809444 25551 raft_consensus.cc:482] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:52:17.809541 25551 raft_consensus.cc:3037] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:17.812088 25551 raft_consensus.cc:504] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0014bb33a7d645a29402c1cc179fdc7d" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35247 } } peers { permanent_uuid: "5cca488f84f34ba29f9601fe6c03363e" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 39635 } } peers { permanent_uuid: "c85c8e1b739e4643921ab60273d16bbf" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37951 } }
I20260621 06:52:17.812644 25551 leader_election.cc:288] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [CANDIDATE]: Term 1 election: Requested vote from peers 0014bb33a7d645a29402c1cc179fdc7d (127.20.120.193:35247), 5cca488f84f34ba29f9601fe6c03363e (127.20.120.194:39635)
I20260621 06:52:17.813050 25215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbbc16b8b974b1cb70a136d4864af7c" candidate_uuid: "c85c8e1b739e4643921ab60273d16bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0014bb33a7d645a29402c1cc179fdc7d"
I20260621 06:52:17.813086 25348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbbc16b8b974b1cb70a136d4864af7c" candidate_uuid: "c85c8e1b739e4643921ab60273d16bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cca488f84f34ba29f9601fe6c03363e"
I20260621 06:52:17.813270 25215 raft_consensus.cc:3037] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:17.813295 25348 raft_consensus.cc:3037] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:17.816452 25348 raft_consensus.cc:2454] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c85c8e1b739e4643921ab60273d16bbf in term 1.
I20260621 06:52:17.816452 25215 raft_consensus.cc:2454] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c85c8e1b739e4643921ab60273d16bbf in term 1.
I20260621 06:52:17.816890 25415 leader_election.cc:302] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0014bb33a7d645a29402c1cc179fdc7d, c85c8e1b739e4643921ab60273d16bbf; no voters:
I20260621 06:52:17.817282 25551 raft_consensus.cc:2781] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:17.817679 25551 raft_consensus.cc:686] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 1 LEADER]: Becoming Leader. State: Replica: c85c8e1b739e4643921ab60273d16bbf, State: Running, Role: LEADER
I20260621 06:52:17.818246 25551 consensus_queue.cc:237] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0014bb33a7d645a29402c1cc179fdc7d" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35247 } } peers { permanent_uuid: "5cca488f84f34ba29f9601fe6c03363e" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 39635 } } peers { permanent_uuid: "c85c8e1b739e4643921ab60273d16bbf" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37951 } }
I20260621 06:52:17.822511 25072 catalog_manager.cc:5495] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf reported cstate change: term changed from 0 to 1, leader changed from <none> to c85c8e1b739e4643921ab60273d16bbf (127.20.120.195). New cstate: current_term: 1 leader_uuid: "c85c8e1b739e4643921ab60273d16bbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0014bb33a7d645a29402c1cc179fdc7d" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cca488f84f34ba29f9601fe6c03363e" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 39635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c85c8e1b739e4643921ab60273d16bbf" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37951 } health_report { overall_health: HEALTHY } } }
W20260621 06:52:17.829672 25394 tablet.cc:2369] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:52:17.846163 25261 tablet.cc:2369] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:52:17.869585 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.864124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.20.120.195:37951)
0621 06:52:17.864350 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:17.864359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:17.864422 (+ 63us) server_negotiation.cc:407] Connection header received
0621 06:52:17.864619 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:17.864623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:17.864714 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:17.864836 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:52:17.865129 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.865832 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.867647 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.868023 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.868116 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.868521 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:52:17.869045 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:52:17.869153 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:17.869222 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 06:52:17.869285 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:52:17.877640 25215 raft_consensus.cc:1261] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d [term 1 FOLLOWER]: Refusing update from remote peer c85c8e1b739e4643921ab60273d16bbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:17.877662 25348 raft_consensus.cc:1261] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e [term 1 FOLLOWER]: Refusing update from remote peer c85c8e1b739e4643921ab60273d16bbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:17.878657 25551 consensus_queue.cc:1035] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0014bb33a7d645a29402c1cc179fdc7d" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:17.878938 25553 consensus_queue.cc:1035] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cca488f84f34ba29f9601fe6c03363e" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 39635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:17.893687 25557 mvcc.cc:205] Tried to move back new op lower bound from 7299173326337474560 to 7299173326107287552. Current Snapshot: MvccSnapshot[applied={T|T < 7299173326337474560}]
I20260621 06:52:17.905754 25070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:17.906069 25070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:17.909578 25070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69f0e4eb82474914b2e0939235322f59: Sending DeleteTablet for 3 replicas of tablet fdbbc16b8b974b1cb70a136d4864af7c
I20260621 06:52:17.910734 25195 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbbc16b8b974b1cb70a136d4864af7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:17 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:58518
I20260621 06:52:17.910965 25328 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbbc16b8b974b1cb70a136d4864af7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:17 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:56438
I20260621 06:52:17.911918 25460 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbbc16b8b974b1cb70a136d4864af7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:17 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:44598
I20260621 06:52:17.912217 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 25133
I20260621 06:52:17.922566 25566 tablet_replica.cc:330] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d: stopping tablet replica
I20260621 06:52:17.923874 25566 raft_consensus.cc:2229] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:17.924539 25566 raft_consensus.cc:2258] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:17.927553 25566 ts_tablet_manager.cc:1905] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 06:52:17 dist-test-slave-gmf2 krb5kdc[25028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:52:17.934402 25568 tablet_replica.cc:330] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf: stopping tablet replica
I20260621 06:52:17.934580 25567 tablet_replica.cc:330] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e: stopping tablet replica
I20260621 06:52:17.934881 25568 raft_consensus.cc:2229] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:52:17.935087 25567 raft_consensus.cc:2229] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:17.935657 25567 raft_consensus.cc:2258] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:17.935663 25568 raft_consensus.cc:2258] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:17.937875 25568 ts_tablet_manager.cc:1905] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:17.937893 25567 ts_tablet_manager.cc:1905] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:17.939352 25566 ts_tablet_manager.cc:1918] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:17.939560 25566 log.cc:1196] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/fdbbc16b8b974b1cb70a136d4864af7c
I20260621 06:52:17.940119 25566 ts_tablet_manager.cc:1939] T fdbbc16b8b974b1cb70a136d4864af7c P 0014bb33a7d645a29402c1cc179fdc7d: Deleting consensus metadata
I20260621 06:52:17.942009 25058 catalog_manager.cc:4841] TS 0014bb33a7d645a29402c1cc179fdc7d (127.20.120.193:35247): tablet fdbbc16b8b974b1cb70a136d4864af7c (table test-table [id=e051841e5ad341f3a5d8912821eea110]) successfully deleted
I20260621 06:52:17.942744 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:17.914683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.20.120.193:35247)
0621 06:52:17.914901 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:17.914910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:17.923179 (+ 8269us) server_negotiation.cc:407] Connection header received
0621 06:52:17.923382 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:17.923387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:17.923468 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:17.923574 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:17.925466 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.926688 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.928552 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:17.929191 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:17.929305 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:17.935081 (+ 5776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:17.935094 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:17.935111 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:17.935148 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:17.939177 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:17.940007 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:17.940015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:17.940019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:17.940192 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:17.940653 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:17.940658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:17.940662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:17.941066 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:17.941393 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:17.942046 (+ 653us) server_negotiation.cc:299] Negotiation successful
0621 06:52:17.942321 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 06:52:17.943689 25255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:17.944464 25567 ts_tablet_manager.cc:1918] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:17.944617 25567 log.cc:1196] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/fdbbc16b8b974b1cb70a136d4864af7c
I20260621 06:52:17.945132 25567 ts_tablet_manager.cc:1939] T fdbbc16b8b974b1cb70a136d4864af7c P 5cca488f84f34ba29f9601fe6c03363e: Deleting consensus metadata
I20260621 06:52:17.945962 25568 ts_tablet_manager.cc:1918] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:17.946141 25568 log.cc:1196] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/fdbbc16b8b974b1cb70a136d4864af7c
I20260621 06:52:17.946795 25568 ts_tablet_manager.cc:1939] T fdbbc16b8b974b1cb70a136d4864af7c P c85c8e1b739e4643921ab60273d16bbf: Deleting consensus metadata
I20260621 06:52:17.946940 25057 catalog_manager.cc:4841] TS 5cca488f84f34ba29f9601fe6c03363e (127.20.120.194:39635): tablet fdbbc16b8b974b1cb70a136d4864af7c (table test-table [id=e051841e5ad341f3a5d8912821eea110]) successfully deleted
I20260621 06:52:17.948801 25058 catalog_manager.cc:4841] TS c85c8e1b739e4643921ab60273d16bbf (127.20.120.195:37951): tablet fdbbc16b8b974b1cb70a136d4864af7c (table test-table [id=e051841e5ad341f3a5d8912821eea110]) successfully deleted
I20260621 06:52:18.072340 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 25133
I20260621 06:52:18.089483 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 25266
Jun 21 06:52:18 dist-test-slave-gmf2 krb5kdc[25028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:52:18.105011 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:18.090280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.20.120.194:39635)
0621 06:52:18.090460 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:18.090467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:18.090613 (+ 146us) server_negotiation.cc:407] Connection header received
0621 06:52:18.090834 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:18.090838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:18.090900 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:18.091011 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:18.092224 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:18.092993 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:18.094360 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:18.094750 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:18.094882 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:18.098849 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:18.098858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:18.098871 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:18.098899 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:18.102245 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:18.102854 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:18.102870 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:18.102873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:18.102987 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:18.103378 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:18.103383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:18.103386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:18.103751 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:18.104034 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:18.104320 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 06:52:18.104531 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 06:52:18.105799 25388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:18.194665 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 25266
I20260621 06:52:18.216383 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 25398
Jun 21 06:52:18 dist-test-slave-gmf2 krb5kdc[25028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:52:18.234283 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:18.217231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.20.120.195:37951)
0621 06:52:18.217439 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:18.217446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:18.217469 (+ 23us) server_negotiation.cc:407] Connection header received
0621 06:52:18.217602 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:18.217613 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:18.217712 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:18.217850 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:18.219099 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:18.220339 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:18.221610 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:18.222475 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:18.222613 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:18.227061 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:18.227075 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:18.227096 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:18.227147 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:18.231318 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:18.231967 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:18.231977 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:18.231981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:18.232177 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:18.232611 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:18.232616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:18.232621 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:18.233062 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:18.233394 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:18.233584 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 06:52:18.233803 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 06:52:18.235106 25520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:18.351953 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 25398
I20260621 06:52:18.370484 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 25041
I20260621 06:52:18.371570 25102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:18.474476 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 25041
2026-06-21T06:52:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4171 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-21T06:52:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:18Z Disabled control of system clock
I20260621 06:52:18.537663 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34037
--webserver_interface=127.20.120.254
--webserver_port=0
--builtin_ntp_servers=127.20.120.212:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:34037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 06:52:18.844452 25577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:18.844712 25577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:18.844791 25577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:18.851967 25577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:18.852069 25577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:18.852109 25577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:18.852165 25577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:18.852206 25577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:18.860814 25577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:34037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:18.862107 25577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:18.863755 25577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:18.874831 25583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:18.874935 25582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:18.874902 25585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:18.875706 25577 server_base.cc:1034] running on GCE node
I20260621 06:52:18.876453 25577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:18.877823 25577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:18.879027 25577 hybrid_clock.cc:648] HybridClock initialized: now 1782024738878987 us; error 35 us; skew 500 ppm
I20260621 06:52:18.879415 25577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:52:18.881665 25577 webserver.cc:460] Webserver started at http://127.20.120.254:38389/ using document root <none> and password file <none>
I20260621 06:52:18.882470 25577 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:18.882586 25577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:18.882925 25577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:18.885558 25577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "8641b0a0b7184a288130a584b45106b3"
format_stamp: "Formatted at 2026-06-21 06:52:18 on dist-test-slave-gmf2"
server_key: "a7b3fe607bee15a0e504df9b2d46f902"
server_key_iv: "4e840ee8f5ec973626c9ca125fb5752e"
server_key_version: "encryptionkey@0"
I20260621 06:52:18.886443 25577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "8641b0a0b7184a288130a584b45106b3"
format_stamp: "Formatted at 2026-06-21 06:52:18 on dist-test-slave-gmf2"
server_key: "a7b3fe607bee15a0e504df9b2d46f902"
server_key_iv: "4e840ee8f5ec973626c9ca125fb5752e"
server_key_version: "encryptionkey@0"
I20260621 06:52:18.891289 25577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 06:52:18.894829 25591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:18.896056 25577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:18.896234 25577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "8641b0a0b7184a288130a584b45106b3"
format_stamp: "Formatted at 2026-06-21 06:52:18 on dist-test-slave-gmf2"
server_key: "a7b3fe607bee15a0e504df9b2d46f902"
server_key_iv: "4e840ee8f5ec973626c9ca125fb5752e"
server_key_version: "encryptionkey@0"
I20260621 06:52:18.896437 25577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:18.901801 25577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:18.902591 25577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:18.902868 25577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:18.919049 25577 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:34037
I20260621 06:52:18.919077 25642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:34037 every 8 connection(s)
I20260621 06:52:18.920251 25577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:18.923135 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 25577
I20260621 06:52:18.923316 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:18.923841 25643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:18.923987 20963 external_mini_cluster.cc:1428] Setting key 8d99d44a51c43f8acf2ef5b1076cd328
I20260621 06:52:18.932530 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:18.925967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.20.120.254:34037)
0621 06:52:18.927082 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:18.927095 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:18.927140 (+ 45us) server_negotiation.cc:407] Connection header received
0621 06:52:18.928687 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:18.928714 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:18.929229 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:18.929652 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:18.930636 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:18.930644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:18.930664 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:18.930679 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:18.930766 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:18.930812 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:18.931026 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:18.931148 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 06:52:18.931383 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":336,"threads_started":1}
I20260621 06:52:18.937497 25643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3: Bootstrap starting.
I20260621 06:52:18.940371 25643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:18.941215 25643 log.cc:824] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:18.943677 25643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3: No bootstrap required, opened a new log
I20260621 06:52:18.950402 25643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8641b0a0b7184a288130a584b45106b3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34037 } }
I20260621 06:52:18.950680 25643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:18.950743 25643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8641b0a0b7184a288130a584b45106b3, State: Initialized, Role: FOLLOWER
I20260621 06:52:18.951220 25643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8641b0a0b7184a288130a584b45106b3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34037 } }
I20260621 06:52:18.951412 25643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:18.951493 25643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:18.951597 25643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:18.953526 25643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8641b0a0b7184a288130a584b45106b3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34037 } }
I20260621 06:52:18.954052 25643 leader_election.cc:302] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8641b0a0b7184a288130a584b45106b3; no voters:
I20260621 06:52:18.954703 25643 leader_election.cc:288] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:18.954846 25648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:18.955533 25648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [term 1 LEADER]: Becoming Leader. State: Replica: 8641b0a0b7184a288130a584b45106b3, State: Running, Role: LEADER
I20260621 06:52:18.956022 25648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8641b0a0b7184a288130a584b45106b3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34037 } }
I20260621 06:52:18.956562 25643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:18.958765 25650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8641b0a0b7184a288130a584b45106b3. Latest consensus state: current_term: 1 leader_uuid: "8641b0a0b7184a288130a584b45106b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8641b0a0b7184a288130a584b45106b3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34037 } } }
I20260621 06:52:18.959074 25650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:18.958871 25649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8641b0a0b7184a288130a584b45106b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8641b0a0b7184a288130a584b45106b3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34037 } } }
I20260621 06:52:18.959235 25649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3 [sys.catalog]: This master's current role is: LEADER
W20260621 06:52:18.970005 25664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 06:52:18.970139 25664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 06:52:18.973097 25655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:18.974776 25655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:18.982693 25655 catalog_manager.cc:1347] Generated new cluster ID: dc9fc83cbd2e438782b205e9b599e41d
I20260621 06:52:18.982806 25655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:19.001892 25655 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:19.002334 25655 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:19.010628 25655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3: Generated new TSK 0
I20260621 06:52:19.011381 25655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:52:19.015398 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34037
--builtin_ntp_servers=127.20.120.212:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 06:52:19.329643 25667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:19.329921 25667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:19.330061 25667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:19.336771 25667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:19.336867 25667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:19.336987 25667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:52:19.345338 25667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:19.346659 25667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:19.348459 25667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:19.361178 25672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:19.361303 25675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:19.361462 25673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:19.362825 25667 server_base.cc:1034] running on GCE node
I20260621 06:52:19.363421 25667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:19.364393 25667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:19.365581 25667 hybrid_clock.cc:648] HybridClock initialized: now 1782024739365548 us; error 33 us; skew 500 ppm
I20260621 06:52:19.365897 25667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:52:19.368024 25667 webserver.cc:460] Webserver started at http://127.20.120.193:39769/ using document root <none> and password file <none>
I20260621 06:52:19.368692 25667 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:19.368811 25667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:19.369148 25667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:19.371924 25667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "9e2a388fb6da47ab9a9323d8a47bfb06"
format_stamp: "Formatted at 2026-06-21 06:52:19 on dist-test-slave-gmf2"
server_key: "688110b95936fc129c83fadf090cc5f6"
server_key_iv: "1067cf870cbf618fd1b75bce4e8942f2"
server_key_version: "encryptionkey@0"
I20260621 06:52:19.372681 25667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "9e2a388fb6da47ab9a9323d8a47bfb06"
format_stamp: "Formatted at 2026-06-21 06:52:19 on dist-test-slave-gmf2"
server_key: "688110b95936fc129c83fadf090cc5f6"
server_key_iv: "1067cf870cbf618fd1b75bce4e8942f2"
server_key_version: "encryptionkey@0"
I20260621 06:52:19.377319 25667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 06:52:19.380538 25681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:19.381763 25667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:52:19.381989 25667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "9e2a388fb6da47ab9a9323d8a47bfb06"
format_stamp: "Formatted at 2026-06-21 06:52:19 on dist-test-slave-gmf2"
server_key: "688110b95936fc129c83fadf090cc5f6"
server_key_iv: "1067cf870cbf618fd1b75bce4e8942f2"
server_key_version: "encryptionkey@0"
I20260621 06:52:19.382192 25667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:19.387219 25667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:19.387939 25667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:19.388211 25667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:19.389024 25667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:19.390295 25667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:19.390405 25667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:19.390527 25667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:19.390607 25667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:19.421190 25667 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:39775
I20260621 06:52:19.421236 25793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:39775 every 8 connection(s)
I20260621 06:52:19.422466 25667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:52:19.423075 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 25667
I20260621 06:52:19.423293 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:52:19.424099 20963 external_mini_cluster.cc:1428] Setting key 42ab3a93731cd638b6a9d0f52326efdc
I20260621 06:52:19.426947 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34037
--builtin_ntp_servers=127.20.120.212:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 06:52:19.433897 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:19.425203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:33505 (local address 127.20.120.254:34037)
0621 06:52:19.425398 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:19.425405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:19.427084 (+ 1679us) server_negotiation.cc:407] Connection header received
0621 06:52:19.429699 (+ 2615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:19.429703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:19.429769 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:19.429844 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:19.432901 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:19.432907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:19.432911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:19.432935 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:19.432978 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:19.432997 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:19.433106 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:19.433451 (+ 345us) server_negotiation.cc:299] Negotiation successful
0621 06:52:19.433528 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 06:52:19.435660 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:19.425797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34037 (local address 127.20.120.193:33505)
0621 06:52:19.426907 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:19.426971 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:19.429431 (+ 2460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:19.430126 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:19.430139 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:19.430273 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:19.430998 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:52:19.431004 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:19.431051 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:52:19.431155 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:19.433198 (+ 2043us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:19.433210 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:19.433219 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 06:52:19.434248 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0621 06:52:19.434456 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":300,"threads_started":1}
I20260621 06:52:19.437601 25794 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34037
I20260621 06:52:19.437932 25794 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:19.438524 25794 heartbeater.cc:511] Master 127.20.120.254:34037 requested a full tablet report, sending...
I20260621 06:52:19.441165 25608 ts_manager.cc:194] Registered new tserver with Master: 9e2a388fb6da47ab9a9323d8a47bfb06 (127.20.120.193:39775)
W20260621 06:52:19.740957 25797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:19.741258 25797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:19.741436 25797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:19.748306 25797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:19.748422 25797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:19.748596 25797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:52:19.757162 25797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:19.758463 25797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:19.760186 25797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:19.773162 25802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:19.773133 25805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:19.773262 25803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:19.773960 25797 server_base.cc:1034] running on GCE node
I20260621 06:52:19.774541 25797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:19.775429 25797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:19.776614 25797 hybrid_clock.cc:648] HybridClock initialized: now 1782024739776574 us; error 48 us; skew 500 ppm
I20260621 06:52:19.776923 25797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:52:19.779083 25797 webserver.cc:460] Webserver started at http://127.20.120.194:39137/ using document root <none> and password file <none>
I20260621 06:52:19.779757 25797 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:19.779861 25797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:19.780189 25797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:19.782841 25797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "d1e54d35cf4b481d94cd5ade594a97d5"
format_stamp: "Formatted at 2026-06-21 06:52:19 on dist-test-slave-gmf2"
server_key: "69128d3be5ee3ac8871984dd8d98fe6e"
server_key_iv: "d3ae97eafbd31f3effb02198fffbb905"
server_key_version: "encryptionkey@0"
I20260621 06:52:19.783604 25797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "d1e54d35cf4b481d94cd5ade594a97d5"
format_stamp: "Formatted at 2026-06-21 06:52:19 on dist-test-slave-gmf2"
server_key: "69128d3be5ee3ac8871984dd8d98fe6e"
server_key_iv: "d3ae97eafbd31f3effb02198fffbb905"
server_key_version: "encryptionkey@0"
I20260621 06:52:19.788905 25797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 06:52:19.792094 25811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:19.793344 25797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 06:52:19.793504 25797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "d1e54d35cf4b481d94cd5ade594a97d5"
format_stamp: "Formatted at 2026-06-21 06:52:19 on dist-test-slave-gmf2"
server_key: "69128d3be5ee3ac8871984dd8d98fe6e"
server_key_iv: "d3ae97eafbd31f3effb02198fffbb905"
server_key_version: "encryptionkey@0"
I20260621 06:52:19.793728 25797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:19.798884 25797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:19.799768 25797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:19.800017 25797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:19.800990 25797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:19.802311 25797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:19.802414 25797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:19.802536 25797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:19.802624 25797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:19.833050 25797 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:35889
I20260621 06:52:19.833094 25923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:35889 every 8 connection(s)
I20260621 06:52:19.834393 25797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:52:19.840582 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 25797
I20260621 06:52:19.840734 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:52:19.841223 20963 external_mini_cluster.cc:1428] Setting key 4338a711cfc410e2ad33aef7a7b2d444
I20260621 06:52:19.843133 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:19.836940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:47415 (local address 127.20.120.254:34037)
0621 06:52:19.837105 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:19.837110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:19.838519 (+ 1409us) server_negotiation.cc:407] Connection header received
0621 06:52:19.840197 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:19.840202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:19.840278 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:19.840405 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:19.841846 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:19.841850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:19.841854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:19.841870 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:19.841901 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:19.841959 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:19.842111 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:19.842835 (+ 724us) server_negotiation.cc:299] Negotiation successful
0621 06:52:19.842892 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 06:52:19.843606 25926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:19.837413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34037 (local address 127.20.120.194:47415)
0621 06:52:19.838361 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:19.838404 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:19.839931 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:19.840630 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:19.840659 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:19.840805 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:19.841524 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:52:19.841526 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:19.841563 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:52:19.841644 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:19.842094 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:19.842104 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:19.842110 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 06:52:19.842326 (+ 216us) client_negotiation.cc:240] Negotiation successful
0621 06:52:19.842517 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":264,"threads_started":1}
I20260621 06:52:19.844090 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34037
--builtin_ntp_servers=127.20.120.212:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 06:52:19.844983 25924 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34037
I20260621 06:52:19.845309 25924 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:19.846072 25924 heartbeater.cc:511] Master 127.20.120.254:34037 requested a full tablet report, sending...
I20260621 06:52:19.847728 25608 ts_manager.cc:194] Registered new tserver with Master: d1e54d35cf4b481d94cd5ade594a97d5 (127.20.120.194:35889)
W20260621 06:52:20.169028 25927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:20.169337 25927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:20.169456 25927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:20.176607 25927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:20.176709 25927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:20.176818 25927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:52:20.185478 25927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:20.186882 25927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:20.188689 25927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:20.201030 25932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:20.201131 25933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:20.201162 25935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:20.202004 25927 server_base.cc:1034] running on GCE node
I20260621 06:52:20.202548 25927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:20.203508 25927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:20.204728 25927 hybrid_clock.cc:648] HybridClock initialized: now 1782024740204679 us; error 39 us; skew 500 ppm
I20260621 06:52:20.205101 25927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:52:20.207512 25927 webserver.cc:460] Webserver started at http://127.20.120.195:43379/ using document root <none> and password file <none>
I20260621 06:52:20.208230 25927 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:20.208317 25927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:20.208686 25927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:20.211620 25927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "4ef6bffd06f048d987758715c3c98eba"
format_stamp: "Formatted at 2026-06-21 06:52:20 on dist-test-slave-gmf2"
server_key: "999f05777071efdbabfe4433e061f985"
server_key_iv: "74638bdb60088d8847fc7572e56d5ac2"
server_key_version: "encryptionkey@0"
I20260621 06:52:20.212426 25927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "4ef6bffd06f048d987758715c3c98eba"
format_stamp: "Formatted at 2026-06-21 06:52:20 on dist-test-slave-gmf2"
server_key: "999f05777071efdbabfe4433e061f985"
server_key_iv: "74638bdb60088d8847fc7572e56d5ac2"
server_key_version: "encryptionkey@0"
I20260621 06:52:20.217447 25927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 06:52:20.221002 25941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:20.222347 25927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:20.222515 25927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "4ef6bffd06f048d987758715c3c98eba"
format_stamp: "Formatted at 2026-06-21 06:52:20 on dist-test-slave-gmf2"
server_key: "999f05777071efdbabfe4433e061f985"
server_key_iv: "74638bdb60088d8847fc7572e56d5ac2"
server_key_version: "encryptionkey@0"
I20260621 06:52:20.222738 25927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:20.228370 25927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:20.229195 25927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:20.229486 25927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:20.230415 25927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:20.231964 25927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:20.232110 25927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:20.232247 25927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:20.232337 25927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:20.264557 25927 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:40723
I20260621 06:52:20.264601 26053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:40723 every 8 connection(s)
I20260621 06:52:20.265877 25927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:52:20.273739 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.268519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:43473 (local address 127.20.120.254:34037)
0621 06:52:20.268701 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:20.268705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:20.270113 (+ 1408us) server_negotiation.cc:407] Connection header received
0621 06:52:20.271742 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:20.271748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:20.271838 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:20.271986 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:20.273162 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:20.273166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:20.273169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:20.273184 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:20.273213 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:20.273223 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:20.273321 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:20.273485 (+ 164us) server_negotiation.cc:299] Negotiation successful
0621 06:52:20.273532 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 06:52:20.274180 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 25927
I20260621 06:52:20.274384 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:52:20.274928 20963 external_mini_cluster.cc:1428] Setting key b3b52f5d5a5bc5f181d46e19ca4bd3af
I20260621 06:52:20.274953 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.269047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34037 (local address 127.20.120.195:43473)
0621 06:52:20.269956 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.270006 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:20.271483 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:20.272178 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:20.272185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:20.272277 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:20.272879 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:52:20.272882 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:20.272915 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:52:20.272998 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:20.273303 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:20.273310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:20.273315 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 06:52:20.273532 (+ 217us) client_negotiation.cc:240] Negotiation successful
0621 06:52:20.273747 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":261,"threads_started":1}
I20260621 06:52:20.276340 26054 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34037
I20260621 06:52:20.276626 26054 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:20.277345 26054 heartbeater.cc:511] Master 127.20.120.254:34037 requested a full tablet report, sending...
I20260621 06:52:20.278638 25608 ts_manager.cc:194] Registered new tserver with Master: 4ef6bffd06f048d987758715c3c98eba (127.20.120.195:40723)
I20260621 06:52:20.289732 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:52:20.296288 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.293963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.20.120.254:34037)
0621 06:52:20.294141 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:20.294148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:20.294739 (+ 591us) server_negotiation.cc:407] Connection header received
0621 06:52:20.294858 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:20.294862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:20.294910 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:20.294984 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:20.295842 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:20.295846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:20.295849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:20.295859 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:20.295882 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:20.295892 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:20.295974 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:20.296042 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 06:52:20.296082 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 06:52:20.299860 25608 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:52:20.306507 25608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:52:20.335706 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.317686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.20.120.195:40723)
0621 06:52:20.330086 (+ 12400us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:20.330100 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:20.331317 (+ 1217us) server_negotiation.cc:407] Connection header received
0621 06:52:20.332303 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:20.332313 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:20.332569 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:20.332707 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:20.334634 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:20.334643 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:20.334656 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:20.334674 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:20.334740 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:20.334760 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:20.334933 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:20.335178 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 06:52:20.335278 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12133,"thread_start_us":12417,"threads_started":1}
I20260621 06:52:20.335762 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.317522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:40723 (local address 127.0.0.1:50986)
0621 06:52:20.331079 (+ 13557us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.331131 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:20.332140 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:20.333354 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:20.333364 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:20.333464 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:20.334314 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:52:20.334318 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:20.334351 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:52:20.334447 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:20.334978 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:20.334999 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:20.335005 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 06:52:20.335184 (+ 179us) client_negotiation.cc:240] Negotiation successful
0621 06:52:20.335284 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13201,"thread_start_us":13008,"threads_started":1}
I20260621 06:52:20.337533 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.317958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:39775 (local address 127.0.0.1:32994)
0621 06:52:20.331864 (+ 13906us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.331879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:20.332010 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:20.335539 (+ 3529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:20.335545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:20.335566 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:20.336390 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:52:20.336393 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:20.336408 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:52:20.336449 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:20.336853 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:20.336858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:20.336863 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 06:52:20.337019 (+ 156us) client_negotiation.cc:240] Negotiation successful
0621 06:52:20.337073 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13761,"thread_start_us":13627,"threads_started":1}
I20260621 06:52:20.337935 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.318370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.20.120.194:35889)
0621 06:52:20.333444 (+ 15074us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:20.333456 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:20.333494 (+ 38us) server_negotiation.cc:407] Connection header received
0621 06:52:20.333580 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:20.333591 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:20.333853 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:20.334088 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:20.336907 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:20.336915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:20.336928 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:20.336956 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:20.337012 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:20.337027 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:20.337198 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:20.337316 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 06:52:20.337409 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14885,"thread_start_us":14757,"threads_started":1}
I20260621 06:52:20.338187 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.318780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.20.120.193:39775)
0621 06:52:20.334252 (+ 15472us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:20.334261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:20.334300 (+ 39us) server_negotiation.cc:407] Connection header received
0621 06:52:20.334366 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:20.334373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:20.334568 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:20.334692 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:20.336618 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:20.336627 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:20.336641 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:20.336667 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:20.336729 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:20.336745 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:20.336890 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:20.337745 (+ 855us) server_negotiation.cc:299] Negotiation successful
0621 06:52:20.337847 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15294,"thread_start_us":15162,"threads_started":1}
I20260621 06:52:20.337796 26069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.318305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:35889 (local address 127.0.0.1:34472)
0621 06:52:20.333040 (+ 14735us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.333058 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:20.333151 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:20.336062 (+ 2911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:20.336068 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:20.336094 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:20.336683 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:52:20.336686 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:20.336710 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:52:20.336747 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:20.337147 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:20.337155 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:20.337161 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 06:52:20.337321 (+ 160us) client_negotiation.cc:240] Negotiation successful
0621 06:52:20.337368 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14632,"thread_start_us":14509,"threads_started":1}
I20260621 06:52:20.340063 25989 tablet_service.cc:1467] Processing CreateTablet for tablet 60c7d03342b3425c921e671f7ca42fd8 (DEFAULT_TABLE table=test-table [id=bb1d1f612abf45f29b3b23647d50d3a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:20.341367 25989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c7d03342b3425c921e671f7ca42fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:20.342120 25729 tablet_service.cc:1467] Processing CreateTablet for tablet 60c7d03342b3425c921e671f7ca42fd8 (DEFAULT_TABLE table=test-table [id=bb1d1f612abf45f29b3b23647d50d3a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:20.342120 25859 tablet_service.cc:1467] Processing CreateTablet for tablet 60c7d03342b3425c921e671f7ca42fd8 (DEFAULT_TABLE table=test-table [id=bb1d1f612abf45f29b3b23647d50d3a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:20.343336 25859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c7d03342b3425c921e671f7ca42fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:20.343336 25729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c7d03342b3425c921e671f7ca42fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:20.352418 26072 tablet_bootstrap.cc:492] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba: Bootstrap starting.
I20260621 06:52:20.356220 26072 tablet_bootstrap.cc:654] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:20.357491 26072 log.cc:824] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:20.357728 26073 tablet_bootstrap.cc:492] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5: Bootstrap starting.
I20260621 06:52:20.362185 26073 tablet_bootstrap.cc:654] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:20.363505 26073 log.cc:824] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:20.364305 26074 tablet_bootstrap.cc:492] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06: Bootstrap starting.
I20260621 06:52:20.364842 26072 tablet_bootstrap.cc:492] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba: No bootstrap required, opened a new log
I20260621 06:52:20.365124 26072 ts_tablet_manager.cc:1397] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260621 06:52:20.366161 26073 tablet_bootstrap.cc:492] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5: No bootstrap required, opened a new log
I20260621 06:52:20.366415 26073 ts_tablet_manager.cc:1397] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260621 06:52:20.367823 26074 tablet_bootstrap.cc:654] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:20.368996 26074 log.cc:824] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:20.373673 26073 raft_consensus.cc:348] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef6bffd06f048d987758715c3c98eba" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40723 } } peers { permanent_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 39775 } } peers { permanent_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35889 } }
I20260621 06:52:20.374130 26073 raft_consensus.cc:374] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:20.374229 26073 raft_consensus.cc:729] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e54d35cf4b481d94cd5ade594a97d5, State: Initialized, Role: FOLLOWER
I20260621 06:52:20.374922 26073 consensus_queue.cc:260] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef6bffd06f048d987758715c3c98eba" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40723 } } peers { permanent_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 39775 } } peers { permanent_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35889 } }
I20260621 06:52:20.376011 25924 heartbeater.cc:503] Master 127.20.120.254:34037 was elected leader, sending a full tablet report...
I20260621 06:52:20.376531 26073 ts_tablet_manager.cc:1428] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260621 06:52:20.376869 26074 tablet_bootstrap.cc:492] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06: No bootstrap required, opened a new log
I20260621 06:52:20.377157 26074 ts_tablet_manager.cc:1397] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260621 06:52:20.377658 26072 raft_consensus.cc:348] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef6bffd06f048d987758715c3c98eba" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40723 } } peers { permanent_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 39775 } } peers { permanent_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35889 } }
I20260621 06:52:20.378120 26072 raft_consensus.cc:374] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:20.378249 26072 raft_consensus.cc:729] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef6bffd06f048d987758715c3c98eba, State: Initialized, Role: FOLLOWER
I20260621 06:52:20.378800 26072 consensus_queue.cc:260] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef6bffd06f048d987758715c3c98eba" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40723 } } peers { permanent_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 39775 } } peers { permanent_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35889 } }
I20260621 06:52:20.379920 26054 heartbeater.cc:503] Master 127.20.120.254:34037 was elected leader, sending a full tablet report...
I20260621 06:52:20.380998 26072 ts_tablet_manager.cc:1428] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260621 06:52:20.385568 26074 raft_consensus.cc:348] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef6bffd06f048d987758715c3c98eba" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40723 } } peers { permanent_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 39775 } } peers { permanent_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35889 } }
I20260621 06:52:20.385998 26074 raft_consensus.cc:374] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:20.386076 26074 raft_consensus.cc:729] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2a388fb6da47ab9a9323d8a47bfb06, State: Initialized, Role: FOLLOWER
I20260621 06:52:20.386608 26074 consensus_queue.cc:260] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef6bffd06f048d987758715c3c98eba" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40723 } } peers { permanent_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 39775 } } peers { permanent_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35889 } }
I20260621 06:52:20.387485 25794 heartbeater.cc:503] Master 127.20.120.254:34037 was elected leader, sending a full tablet report...
I20260621 06:52:20.387902 26074 ts_tablet_manager.cc:1428] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260621 06:52:20.417455 26078 raft_consensus.cc:482] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:52:20.417784 26078 raft_consensus.cc:504] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef6bffd06f048d987758715c3c98eba" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40723 } } peers { permanent_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 39775 } } peers { permanent_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35889 } }
I20260621 06:52:20.419495 26078 leader_election.cc:288] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ef6bffd06f048d987758715c3c98eba (127.20.120.195:40723), 9e2a388fb6da47ab9a9323d8a47bfb06 (127.20.120.193:39775)
I20260621 06:52:20.422971 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.419891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:51291 (local address 127.20.120.195:40723)
0621 06:52:20.420063 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:20.420069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:20.420951 (+ 882us) server_negotiation.cc:407] Connection header received
0621 06:52:20.421145 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:20.421149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:20.421196 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:20.421290 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:20.422347 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:20.422353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:20.422357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:20.422382 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:20.422412 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:20.422423 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:20.422550 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:20.422640 (+ 90us) server_negotiation.cc:299] Negotiation successful
0621 06:52:20.422701 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 06:52:20.423099 26081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.419950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:40723 (local address 127.20.120.194:51291)
0621 06:52:20.420844 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.420859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:20.421002 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:20.421323 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:20.421330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:20.421350 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:20.422121 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:52:20.422126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:20.422144 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:52:20.422196 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:20.422512 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:20.422516 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:20.422520 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 06:52:20.422660 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 06:52:20.422706 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":355,"threads_started":1}
I20260621 06:52:20.423237 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.419928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:40637 (local address 127.20.120.193:39775)
0621 06:52:20.420076 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:20.420083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:20.420994 (+ 911us) server_negotiation.cc:407] Connection header received
0621 06:52:20.421223 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:20.421228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:20.421279 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:20.421341 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:20.422510 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:20.422514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:20.422518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:20.422538 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:20.422569 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:20.422582 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:20.422664 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:20.422895 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 06:52:20.422954 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 06:52:20.423988 26082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.419951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:39775 (local address 127.20.120.194:40637)
0621 06:52:20.420902 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.420920 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:20.421074 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:20.421526 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:20.421532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:20.421558 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:20.422306 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:52:20.422310 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:20.422324 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:52:20.422364 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:20.422764 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:20.422769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:20.422773 (+ 4us) client_negotiation.cc:769] Sending connection context
0621 06:52:20.423533 (+ 760us) client_negotiation.cc:240] Negotiation successful
0621 06:52:20.423581 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":140,"thread_start_us":456,"threads_started":1}
I20260621 06:52:20.424101 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c7d03342b3425c921e671f7ca42fd8" candidate_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ef6bffd06f048d987758715c3c98eba" is_pre_election: true
I20260621 06:52:20.424701 26009 raft_consensus.cc:2454] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e54d35cf4b481d94cd5ade594a97d5 in term 0.
W20260621 06:52:20.424897 25795 tablet.cc:2369] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:52:20.425002 25749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c7d03342b3425c921e671f7ca42fd8" candidate_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06" is_pre_election: true
I20260621 06:52:20.425419 25812 leader_election.cc:302] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ef6bffd06f048d987758715c3c98eba, d1e54d35cf4b481d94cd5ade594a97d5; no voters:
I20260621 06:52:20.425562 25749 raft_consensus.cc:2454] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e54d35cf4b481d94cd5ade594a97d5 in term 0.
I20260621 06:52:20.425841 26078 raft_consensus.cc:2781] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:52:20.426017 26078 raft_consensus.cc:482] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:52:20.426136 26078 raft_consensus.cc:3037] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:20.429184 26078 raft_consensus.cc:504] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef6bffd06f048d987758715c3c98eba" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40723 } } peers { permanent_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 39775 } } peers { permanent_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35889 } }
I20260621 06:52:20.429932 26078 leader_election.cc:288] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [CANDIDATE]: Term 1 election: Requested vote from peers 4ef6bffd06f048d987758715c3c98eba (127.20.120.195:40723), 9e2a388fb6da47ab9a9323d8a47bfb06 (127.20.120.193:39775)
I20260621 06:52:20.430380 25749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c7d03342b3425c921e671f7ca42fd8" candidate_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06"
I20260621 06:52:20.430578 25749 raft_consensus.cc:3037] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:20.433820 25749 raft_consensus.cc:2454] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e54d35cf4b481d94cd5ade594a97d5 in term 1.
I20260621 06:52:20.434211 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c7d03342b3425c921e671f7ca42fd8" candidate_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ef6bffd06f048d987758715c3c98eba"
I20260621 06:52:20.434322 25815 leader_election.cc:302] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e2a388fb6da47ab9a9323d8a47bfb06, d1e54d35cf4b481d94cd5ade594a97d5; no voters:
I20260621 06:52:20.434461 26009 raft_consensus.cc:3037] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:20.434675 26078 raft_consensus.cc:2781] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:20.435213 26078 raft_consensus.cc:686] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 1 LEADER]: Becoming Leader. State: Replica: d1e54d35cf4b481d94cd5ade594a97d5, State: Running, Role: LEADER
I20260621 06:52:20.435755 26078 consensus_queue.cc:237] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef6bffd06f048d987758715c3c98eba" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40723 } } peers { permanent_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 39775 } } peers { permanent_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35889 } }
I20260621 06:52:20.437711 26009 raft_consensus.cc:2454] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e54d35cf4b481d94cd5ade594a97d5 in term 1.
I20260621 06:52:20.440752 25607 catalog_manager.cc:5495] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e54d35cf4b481d94cd5ade594a97d5 (127.20.120.194). New cstate: current_term: 1 leader_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef6bffd06f048d987758715c3c98eba" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 39775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e54d35cf4b481d94cd5ade594a97d5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 35889 } health_report { overall_health: HEALTHY } } }
I20260621 06:52:20.461762 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.459196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.20.120.194:35889)
0621 06:52:20.459403 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:20.459410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:20.459522 (+ 112us) server_negotiation.cc:407] Connection header received
0621 06:52:20.459767 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:20.459771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:20.459852 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:20.459967 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:20.461114 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:20.461120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:20.461125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:20.461142 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:20.461203 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:20.461217 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:20.461329 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:20.461421 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 06:52:20.461467 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 06:52:20.469755 26009 raft_consensus.cc:1261] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba [term 1 FOLLOWER]: Refusing update from remote peer d1e54d35cf4b481d94cd5ade594a97d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:20.469754 25749 raft_consensus.cc:1261] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06 [term 1 FOLLOWER]: Refusing update from remote peer d1e54d35cf4b481d94cd5ade594a97d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:20.470719 26083 consensus_queue.cc:1035] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e2a388fb6da47ab9a9323d8a47bfb06" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:20.471027 26078 consensus_queue.cc:1035] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ef6bffd06f048d987758715c3c98eba" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:20.481630 26086 mvcc.cc:205] Tried to move back new op lower bound from 7299173336954908672 to 7299173336829878272. Current Snapshot: MvccSnapshot[applied={T|T < 7299173336954908672}]
I20260621 06:52:20.495344 25608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:20.495632 25608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:20.499015 25608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8641b0a0b7184a288130a584b45106b3: Sending DeleteTablet for 3 replicas of tablet 60c7d03342b3425c921e671f7ca42fd8
I20260621 06:52:20.500137 25989 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c7d03342b3425c921e671f7ca42fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:20 UTC) from {username='slave'} at 127.0.0.1:50986
I20260621 06:52:20.500334 25859 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c7d03342b3425c921e671f7ca42fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:20 UTC) from {username='slave'} at 127.0.0.1:34472
I20260621 06:52:20.501099 25729 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c7d03342b3425c921e671f7ca42fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:20 UTC) from {username='slave'} at 127.0.0.1:32994
I20260621 06:52:20.502079 26097 tablet_replica.cc:330] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06: stopping tablet replica
I20260621 06:52:20.503991 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 25667
I20260621 06:52:20.514249 26097 raft_consensus.cc:2229] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:20.514585 26096 tablet_replica.cc:330] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5: stopping tablet replica
I20260621 06:52:20.515308 26096 raft_consensus.cc:2229] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:52:20.515960 26097 raft_consensus.cc:2258] T 60c7d03342b3425c921e671f7ca42fd8 P 9e2a388fb6da47ab9a9323d8a47bfb06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:20.516048 26096 raft_consensus.cc:2258] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:20.517843 26095 tablet_replica.cc:330] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba: stopping tablet replica
I20260621 06:52:20.518797 26095 raft_consensus.cc:2229] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:20.519311 26095 raft_consensus.cc:2258] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:20.521602 26095 ts_tablet_manager.cc:1905] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:20.523367 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.505361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.20.120.193:39775)
0621 06:52:20.505593 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:20.505603 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:20.520317 (+ 14714us) server_negotiation.cc:407] Connection header received
0621 06:52:20.520610 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:20.520616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:20.520920 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:20.521020 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:20.522612 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:20.522618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:20.522623 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:20.522644 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:20.522683 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:20.522710 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:20.522806 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:20.523036 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 06:52:20.523099 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:52:20.524716 25789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:20.527333 26096 ts_tablet_manager.cc:1905] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:20.529120 26095 ts_tablet_manager.cc:1918] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:20.529301 26095 log.cc:1196] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/60c7d03342b3425c921e671f7ca42fd8
I20260621 06:52:20.529809 26095 ts_tablet_manager.cc:1939] T 60c7d03342b3425c921e671f7ca42fd8 P 4ef6bffd06f048d987758715c3c98eba: Deleting consensus metadata
I20260621 06:52:20.531678 25592 catalog_manager.cc:4841] TS 4ef6bffd06f048d987758715c3c98eba (127.20.120.195:40723): tablet 60c7d03342b3425c921e671f7ca42fd8 (table test-table [id=bb1d1f612abf45f29b3b23647d50d3a6]) successfully deleted
I20260621 06:52:20.533834 26096 ts_tablet_manager.cc:1918] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:20.536907 26096 log.cc:1196] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/60c7d03342b3425c921e671f7ca42fd8
I20260621 06:52:20.537616 26096 ts_tablet_manager.cc:1939] T 60c7d03342b3425c921e671f7ca42fd8 P d1e54d35cf4b481d94cd5ade594a97d5: Deleting consensus metadata
I20260621 06:52:20.539928 25593 catalog_manager.cc:4841] TS d1e54d35cf4b481d94cd5ade594a97d5 (127.20.120.194:35889): tablet 60c7d03342b3425c921e671f7ca42fd8 (table test-table [id=bb1d1f612abf45f29b3b23647d50d3a6]) successfully deleted
I20260621 06:52:20.675060 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 25667
W20260621 06:52:20.689649 25595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.120.193:39775 (error 108)
I20260621 06:52:20.690694 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 25797
I20260621 06:52:20.692070 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.691488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:39775 (local address 127.0.0.1:33024)
0621 06:52:20.691738 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.691908 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:39775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260621 06:52:20.692317 25595 catalog_manager.cc:4584] TS 9e2a388fb6da47ab9a9323d8a47bfb06 (127.20.120.193:39775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60c7d03342b3425c921e671f7ca42fd8: Network error: Client connection negotiation failed: client connection to 127.20.120.193:39775: connect: Connection refused (error 111)
I20260621 06:52:20.694015 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.691498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.20.120.194:35889)
0621 06:52:20.692002 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:20.692006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:20.692020 (+ 14us) server_negotiation.cc:407] Connection header received
0621 06:52:20.692093 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:20.692096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:20.692176 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:20.692268 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:20.693319 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:20.693324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:20.693329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:20.693345 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:20.693385 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:20.693396 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:20.693539 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:20.693637 (+ 98us) server_negotiation.cc:299] Negotiation successful
0621 06:52:20.693692 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407}
I20260621 06:52:20.694653 25919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:20.746105 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.745575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:39775 (local address 127.0.0.1:33040)
0621 06:52:20.745796 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.745883 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:39775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260621 06:52:20.747097 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.746756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:39775 (local address 127.0.0.1:33050)
0621 06:52:20.746902 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.746993 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:39775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 06:52:20.800343 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 25797
I20260621 06:52:20.808156 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.807588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:39775 (local address 127.0.0.1:33062)
0621 06:52:20.807831 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.807945 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:39775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260621 06:52:20.809185 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.808793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:39775 (local address 127.0.0.1:33066)
0621 06:52:20.808964 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.809049 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:39775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 06:52:20.817653 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 25927
I20260621 06:52:20.820176 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.818384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.20.120.195:40723)
0621 06:52:20.818554 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:20.818558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:20.818574 (+ 16us) server_negotiation.cc:407] Connection header received
0621 06:52:20.818710 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:20.818713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:20.818778 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:20.818857 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:20.819652 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:20.819655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:20.819658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:20.819669 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:20.819691 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:20.819700 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:20.819790 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:20.819853 (+ 63us) server_negotiation.cc:299] Negotiation successful
0621 06:52:20.819893 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 06:52:20.820732 26049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:20.889518 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.888985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:39775 (local address 127.0.0.1:33068)
0621 06:52:20.889240 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.889346 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:39775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260621 06:52:20.890544 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:20.890148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:39775 (local address 127.0.0.1:33072)
0621 06:52:20.890332 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:20.890417 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:39775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260621 06:52:20.906651 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 25927
I20260621 06:52:20.922999 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 25577
I20260621 06:52:20.924121 25638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:21.028781 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 25577
2026-06-21T06:52:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2554 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260621 06:52:21.069317 20963 mini_oidc.cc:152] Starting JWKS server
I20260621 06:52:21.071918 20963 webserver.cc:460] Webserver started at https://127.0.0.1:46383/ using document root <none> and password file <none>
I20260621 06:52:21.072228 20963 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 06:52:21.072804 20963 webserver.cc:460] Webserver started at http://127.0.0.1:46547/ using document root <none> and password file <none>
2026-06-21T06:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:21Z Disabled control of system clock
I20260621 06:52:21.098055 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:46823
--webserver_interface=127.20.120.254
--webserver_port=0
--builtin_ntp_servers=127.20.120.212:40621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:46823
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46383/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/testchainca.cert with env {}
W20260621 06:52:21.406128 26109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:21.406375 26109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:21.406455 26109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:21.413371 26109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:21.413460 26109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:21.413496 26109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:21.413542 26109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:21.413575 26109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 06:52:21.413630 26109 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 06:52:21.413668 26109 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46383/jwks.json
I20260621 06:52:21.422838 26109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:46823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:46823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46383/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:21.423986 26109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:21.425151 26109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:21.436838 26114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:21.436913 26115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:21.437252 26117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:21.438549 26109 server_base.cc:1034] running on GCE node
I20260621 06:52:21.439543 26109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:21.441341 26109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:21.442600 26109 hybrid_clock.cc:648] HybridClock initialized: now 1782024741442479 us; error 86 us; skew 500 ppm
I20260621 06:52:21.443114 26109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:52:21.450006 26109 webserver.cc:460] Webserver started at http://127.20.120.254:41205/ using document root <none> and password file <none>
I20260621 06:52:21.450740 26109 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:21.450830 26109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:21.451155 26109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:21.454171 26109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "c0c609d51c844672bc6bbf03df959fe7"
format_stamp: "Formatted at 2026-06-21 06:52:21 on dist-test-slave-gmf2"
server_key: "ea1ee0e24ee4d4448f8b5b5639b457f3"
server_key_iv: "4ff78ddb30087b11cfae831e2fa53acc"
server_key_version: "encryptionkey@0"
I20260621 06:52:21.454901 26109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "c0c609d51c844672bc6bbf03df959fe7"
format_stamp: "Formatted at 2026-06-21 06:52:21 on dist-test-slave-gmf2"
server_key: "ea1ee0e24ee4d4448f8b5b5639b457f3"
server_key_iv: "4ff78ddb30087b11cfae831e2fa53acc"
server_key_version: "encryptionkey@0"
I20260621 06:52:21.459728 26109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 06:52:21.463037 26123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:21.464455 26109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:21.464582 26109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "c0c609d51c844672bc6bbf03df959fe7"
format_stamp: "Formatted at 2026-06-21 06:52:21 on dist-test-slave-gmf2"
server_key: "ea1ee0e24ee4d4448f8b5b5639b457f3"
server_key_iv: "4ff78ddb30087b11cfae831e2fa53acc"
server_key_version: "encryptionkey@0"
I20260621 06:52:21.464738 26109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:21.490111 26109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:21.490904 26109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:21.491163 26109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:21.508374 26109 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:46823
I20260621 06:52:21.508397 26174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:46823 every 8 connection(s)
I20260621 06:52:21.509670 26109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:21.513386 26175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:21.514946 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 26109
I20260621 06:52:21.515156 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:21.515769 20963 external_mini_cluster.cc:1428] Setting key c034cac864cefe6ea5a1717c139e7dd9
I20260621 06:52:21.526850 26175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7: Bootstrap starting.
I20260621 06:52:21.530772 26175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:21.531656 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:21.517732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.20.120.254:46823)
0621 06:52:21.518882 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:21.518897 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:21.518950 (+ 53us) server_negotiation.cc:407] Connection header received
0621 06:52:21.521195 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:21.521221 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:21.521821 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:21.522281 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:21.522834 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:21.524859 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:21.526366 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:21.527405 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:21.527590 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:21.528704 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:21.528720 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:21.528744 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:21.529479 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:21.529593 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:21.529618 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:21.529739 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:21.529953 (+ 214us) server_negotiation.cc:299] Negotiation successful
0621 06:52:21.530280 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":401,"threads_started":1}
I20260621 06:52:21.532107 26175 log.cc:824] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:21.535254 26175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7: No bootstrap required, opened a new log
I20260621 06:52:21.535523 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:52:21.542672 26175 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c609d51c844672bc6bbf03df959fe7" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46823 } }
I20260621 06:52:21.542995 26175 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:21.543088 26175 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c609d51c844672bc6bbf03df959fe7, State: Initialized, Role: FOLLOWER
I20260621 06:52:21.543663 26175 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c609d51c844672bc6bbf03df959fe7" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46823 } }
I20260621 06:52:21.543919 26175 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:21.544044 26175 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:21.544212 26175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:21.546273 26175 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c609d51c844672bc6bbf03df959fe7" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46823 } }
I20260621 06:52:21.546756 26175 leader_election.cc:302] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0c609d51c844672bc6bbf03df959fe7; no voters:
I20260621 06:52:21.547329 26175 leader_election.cc:288] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:21.547461 26180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:21.548126 26180 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [term 1 LEADER]: Becoming Leader. State: Replica: c0c609d51c844672bc6bbf03df959fe7, State: Running, Role: LEADER
I20260621 06:52:21.548640 26180 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c609d51c844672bc6bbf03df959fe7" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46823 } }
I20260621 06:52:21.549362 26175 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:21.551343 26182 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0c609d51c844672bc6bbf03df959fe7. Latest consensus state: current_term: 1 leader_uuid: "c0c609d51c844672bc6bbf03df959fe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c609d51c844672bc6bbf03df959fe7" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46823 } } }
I20260621 06:52:21.551637 26182 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:21.551452 26181 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0c609d51c844672bc6bbf03df959fe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c609d51c844672bc6bbf03df959fe7" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46823 } } }
I20260621 06:52:21.551831 26181 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:21.554574 26186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:21.560734 26186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:21.568686 26186 catalog_manager.cc:1347] Generated new cluster ID: 4aef444336224381835a55dfaa2449a5
I20260621 06:52:21.568833 26186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:21.586903 26186 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:21.588776 26186 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:21.601030 26186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0c609d51c844672bc6bbf03df959fe7: Generated new TSK 0
I20260621 06:52:21.602195 26186 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 06:52:21.644309 26205 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260621 06:52:21.644466 26205 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 06:52:21.645409 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:21.618717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.20.120.254:46823)
0621 06:52:21.618912 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:21.618920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:21.619338 (+ 418us) server_negotiation.cc:407] Connection header received
0621 06:52:21.619652 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:21.619656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:21.619714 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:21.619858 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0621 06:52:21.620381 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:21.621044 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:21.622462 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:21.622916 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:21.623040 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:21.623507 (+ 467us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 06:52:21.644698 (+ 21191us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 06:52:21.645096 (+ 398us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38552: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260621 06:52:21.645527 26178 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38552: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260621 06:52:21.646852 26204 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:21.618654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:46823 (local address 127.0.0.1:38552)
0621 06:52:21.619234 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:21.619249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:21.619444 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:21.619887 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:21.619893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:21.619904 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0621 06:52:21.620214 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:21.620240 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:21.621172 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:21.621180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:21.622304 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:21.622314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:21.623095 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:21.623104 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:21.623218 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:21.623267 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 06:52:21.645011 (+ 21744us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 06:52:21.645328 (+ 317us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.120.254:46823: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":324,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.120.254:46823: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260621 06:52:21.648535 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 26109
I20260621 06:52:21.649716 26170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:21.738179 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 26109
2026-06-21T06:52:21Z chronyd exiting
I20260621 06:52:21.787866 20963 test_util.cc:175] -----------------------------------------------
I20260621 06:52:21.788002 20963 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782024700637160-20963-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (722 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260621 06:52:21.791515 20963 mini_oidc.cc:152] Starting JWKS server
I20260621 06:52:21.793455 20963 webserver.cc:460] Webserver started at https://127.0.0.1:43703/ using document root <none> and password file <none>
I20260621 06:52:21.793686 20963 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 06:52:21.794340 20963 webserver.cc:460] Webserver started at http://127.0.0.1:39255/ using document root <none> and password file <none>
2026-06-21T06:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:21Z Disabled control of system clock
I20260621 06:52:21.818926 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:43941
--webserver_interface=127.20.120.254
--webserver_port=0
--builtin_ntp_servers=127.20.120.212:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:43941
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43703/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/testchainca.cert with env {}
W20260621 06:52:22.133088 26212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:22.133515 26212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:22.133720 26212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:22.141301 26212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:22.141479 26212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:22.141580 26212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:22.141673 26212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:22.141770 26212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 06:52:22.141892 26212 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 06:52:22.142006 26212 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43703/jwks.json
I20260621 06:52:22.152820 26212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:43941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:43941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43703/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:22.154258 26212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:22.155607 26212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:22.166855 26217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:22.166908 26218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:22.166908 26220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:22.167958 26212 server_base.cc:1034] running on GCE node
I20260621 06:52:22.168769 26212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:22.170253 26212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:22.171491 26212 hybrid_clock.cc:648] HybridClock initialized: now 1782024742171457 us; error 32 us; skew 500 ppm
I20260621 06:52:22.171856 26212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:52:22.174095 26212 webserver.cc:460] Webserver started at http://127.20.120.254:40507/ using document root <none> and password file <none>
I20260621 06:52:22.174817 26212 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:22.174950 26212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:22.175304 26212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:22.178153 26212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "05abca71976d43fab27a03c135ff2063"
format_stamp: "Formatted at 2026-06-21 06:52:22 on dist-test-slave-gmf2"
server_key: "0b83109ea073536d49a8ed219f999a6f"
server_key_iv: "79a6b652ad54c3d41f3cdf314b7e1912"
server_key_version: "encryptionkey@0"
I20260621 06:52:22.178915 26212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "05abca71976d43fab27a03c135ff2063"
format_stamp: "Formatted at 2026-06-21 06:52:22 on dist-test-slave-gmf2"
server_key: "0b83109ea073536d49a8ed219f999a6f"
server_key_iv: "79a6b652ad54c3d41f3cdf314b7e1912"
server_key_version: "encryptionkey@0"
I20260621 06:52:22.183751 26212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 06:52:22.186836 26226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:22.188133 26212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:52:22.188318 26212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "05abca71976d43fab27a03c135ff2063"
format_stamp: "Formatted at 2026-06-21 06:52:22 on dist-test-slave-gmf2"
server_key: "0b83109ea073536d49a8ed219f999a6f"
server_key_iv: "79a6b652ad54c3d41f3cdf314b7e1912"
server_key_version: "encryptionkey@0"
I20260621 06:52:22.188529 26212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:22.209189 26212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:22.210134 26212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:22.210445 26212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:22.228051 26212 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:43941
I20260621 06:52:22.228070 26277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:43941 every 8 connection(s)
I20260621 06:52:22.229394 26212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:22.233034 26278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:22.235893 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 26212
I20260621 06:52:22.236089 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:22.236922 20963 external_mini_cluster.cc:1428] Setting key 21a93ab48a5979476382c70bb5b3b045
I20260621 06:52:22.245314 26278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063: Bootstrap starting.
I20260621 06:52:22.248968 26278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:22.250118 26278 log.cc:824] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:22.250334 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:22.238894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.20.120.254:43941)
0621 06:52:22.239840 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:22.239850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:22.239890 (+ 40us) server_negotiation.cc:407] Connection header received
0621 06:52:22.241577 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:22.241599 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:22.242152 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:22.242494 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:22.242997 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:22.244254 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:22.246043 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:22.246695 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:22.246837 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:22.248072 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:22.248078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:22.248091 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:22.248609 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:22.248693 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:22.248712 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:22.248791 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:22.249063 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 06:52:22.249337 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":354,"threads_started":1}
I20260621 06:52:22.253103 26278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063: No bootstrap required, opened a new log
I20260621 06:52:22.254014 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:52:22.261827 26278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05abca71976d43fab27a03c135ff2063" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 43941 } }
I20260621 06:52:22.262250 26278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:22.262331 26278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05abca71976d43fab27a03c135ff2063, State: Initialized, Role: FOLLOWER
I20260621 06:52:22.262836 26278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05abca71976d43fab27a03c135ff2063" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 43941 } }
I20260621 06:52:22.263029 26278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:22.263124 26278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:22.263236 26278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:22.265312 26278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05abca71976d43fab27a03c135ff2063" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 43941 } }
I20260621 06:52:22.265758 26278 leader_election.cc:302] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05abca71976d43fab27a03c135ff2063; no voters:
I20260621 06:52:22.266398 26278 leader_election.cc:288] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:22.266585 26283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:22.267752 26283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [term 1 LEADER]: Becoming Leader. State: Replica: 05abca71976d43fab27a03c135ff2063, State: Running, Role: LEADER
I20260621 06:52:22.268249 26283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05abca71976d43fab27a03c135ff2063" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 43941 } }
I20260621 06:52:22.268641 26278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:22.271365 26284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05abca71976d43fab27a03c135ff2063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05abca71976d43fab27a03c135ff2063" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 43941 } } }
I20260621 06:52:22.271787 26284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:22.271675 26285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05abca71976d43fab27a03c135ff2063. Latest consensus state: current_term: 1 leader_uuid: "05abca71976d43fab27a03c135ff2063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05abca71976d43fab27a03c135ff2063" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 43941 } } }
I20260621 06:52:22.272018 26285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:22.272837 26292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:22.278529 26292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:22.287424 26292 catalog_manager.cc:1347] Generated new cluster ID: e1554d3148b44f9eb0d3d5a897217197
I20260621 06:52:22.287549 26292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:22.298836 26292 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:22.300101 26292 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:22.308337 26292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05abca71976d43fab27a03c135ff2063: Generated new TSK 0
I20260621 06:52:22.309094 26292 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 06:52:22.339468 26308 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260621 06:52:22.339607 26308 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 06:52:22.340909 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:22.321018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.20.120.254:43941)
0621 06:52:22.321210 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:22.321216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:22.321621 (+ 405us) server_negotiation.cc:407] Connection header received
0621 06:52:22.321825 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:22.321831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:22.321953 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:22.322114 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0621 06:52:22.322556 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:22.323680 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:22.325022 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:22.325652 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:22.325778 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:22.326253 (+ 475us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 06:52:22.340184 (+ 13931us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 06:52:22.340603 (+ 419us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":62}
W20260621 06:52:22.341037 26281 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260621 06:52:22.340960 26307 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:22.320894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:43941 (local address 127.0.0.1:32858)
0621 06:52:22.321523 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:22.321538 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:22.321654 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:22.322125 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:22.322128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:22.322138 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0621 06:52:22.322380 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:22.322388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:22.323824 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:22.323827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:22.324883 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:22.324893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:22.325790 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:22.325793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:22.325991 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:22.326024 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 06:52:22.340440 (+ 14416us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 06:52:22.340656 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.120.254:43941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":369,"threads_started":1}
I20260621 06:52:22.342660 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 26212
I20260621 06:52:22.343850 26273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:22.435245 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 26212
2026-06-21T06:52:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (700 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260621 06:52:22.492187 20963 mini_oidc.cc:152] Starting JWKS server
I20260621 06:52:22.494076 20963 webserver.cc:460] Webserver started at https://127.0.0.1:36143/ using document root <none> and password file <none>
I20260621 06:52:22.494272 20963 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 06:52:22.494794 20963 webserver.cc:460] Webserver started at http://127.0.0.1:42967/ using document root <none> and password file <none>
2026-06-21T06:52:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:22Z Disabled control of system clock
I20260621 06:52:22.520393 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:37575
--webserver_interface=127.20.120.254
--webserver_port=0
--builtin_ntp_servers=127.20.120.212:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:37575
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36143/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 06:52:22.829820 26315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:22.830214 26315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:22.830366 26315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:22.837428 26315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:22.837575 26315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:22.837666 26315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:22.837772 26315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:22.837865 26315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 06:52:22.837968 26315 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 06:52:22.838029 26315 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36143/jwks.json
I20260621 06:52:22.847179 26315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:37575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:37575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36143/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:22.848518 26315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:22.850346 26315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:22.862267 26323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:22.862372 26321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:22.862308 26320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:22.867352 26315 server_base.cc:1034] running on GCE node
I20260621 06:52:22.868196 26315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:22.869622 26315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:22.870862 26315 hybrid_clock.cc:648] HybridClock initialized: now 1782024742870823 us; error 55 us; skew 500 ppm
I20260621 06:52:22.871219 26315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:52:22.873430 26315 webserver.cc:460] Webserver started at http://127.20.120.254:32901/ using document root <none> and password file <none>
I20260621 06:52:22.874208 26315 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:22.874333 26315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:22.874636 26315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:22.877456 26315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "fa2d5fbaf93948caaf5f57b3f1b1c6c0"
format_stamp: "Formatted at 2026-06-21 06:52:22 on dist-test-slave-gmf2"
server_key: "bde033ba8f00e960be42acd3d67e3f02"
server_key_iv: "c3e06522704d579588519650fc6886c9"
server_key_version: "encryptionkey@0"
I20260621 06:52:22.878281 26315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "fa2d5fbaf93948caaf5f57b3f1b1c6c0"
format_stamp: "Formatted at 2026-06-21 06:52:22 on dist-test-slave-gmf2"
server_key: "bde033ba8f00e960be42acd3d67e3f02"
server_key_iv: "c3e06522704d579588519650fc6886c9"
server_key_version: "encryptionkey@0"
I20260621 06:52:22.883077 26315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 06:52:22.886293 26329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:22.887596 26315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:52:22.887775 26315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "fa2d5fbaf93948caaf5f57b3f1b1c6c0"
format_stamp: "Formatted at 2026-06-21 06:52:22 on dist-test-slave-gmf2"
server_key: "bde033ba8f00e960be42acd3d67e3f02"
server_key_iv: "c3e06522704d579588519650fc6886c9"
server_key_version: "encryptionkey@0"
I20260621 06:52:22.887975 26315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:22.910511 26315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:22.911340 26315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:22.911670 26315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:22.930670 26315 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:37575
I20260621 06:52:22.930689 26380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:37575 every 8 connection(s)
I20260621 06:52:22.932165 26315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:22.935813 26381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:22.938171 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 26315
I20260621 06:52:22.938400 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:22.939164 20963 external_mini_cluster.cc:1428] Setting key 97ca1990a52ac34a946886f9fc541528
I20260621 06:52:22.947867 26381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0: Bootstrap starting.
I20260621 06:52:22.951295 26381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:22.952494 26381 log.cc:824] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:22.953438 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:22.941170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.20.120.254:37575)
0621 06:52:22.942260 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:22.942271 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:22.942312 (+ 41us) server_negotiation.cc:407] Connection header received
0621 06:52:22.944006 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:22.944028 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:22.944493 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:22.944836 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:22.945395 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:22.946810 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:22.948646 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:22.949325 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:22.949467 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:22.950710 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:22.950718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:22.950731 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:22.951262 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:22.951379 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:22.951444 (+ 65us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:22.951554 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:22.951805 (+ 251us) server_negotiation.cc:299] Negotiation successful
0621 06:52:22.952118 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":469,"threads_started":1}
I20260621 06:52:22.955503 26381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0: No bootstrap required, opened a new log
I20260621 06:52:22.957360 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:52:22.963546 26381 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2d5fbaf93948caaf5f57b3f1b1c6c0" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37575 } }
I20260621 06:52:22.963848 26381 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:22.963909 26381 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa2d5fbaf93948caaf5f57b3f1b1c6c0, State: Initialized, Role: FOLLOWER
I20260621 06:52:22.964336 26381 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2d5fbaf93948caaf5f57b3f1b1c6c0" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37575 } }
I20260621 06:52:22.964527 26381 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:22.964625 26381 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:22.964722 26381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:22.966710 26381 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2d5fbaf93948caaf5f57b3f1b1c6c0" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37575 } }
I20260621 06:52:22.967218 26381 leader_election.cc:302] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa2d5fbaf93948caaf5f57b3f1b1c6c0; no voters:
I20260621 06:52:22.967826 26381 leader_election.cc:288] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:22.967985 26386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:22.968917 26386 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [term 1 LEADER]: Becoming Leader. State: Replica: fa2d5fbaf93948caaf5f57b3f1b1c6c0, State: Running, Role: LEADER
I20260621 06:52:22.969537 26386 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2d5fbaf93948caaf5f57b3f1b1c6c0" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37575 } }
I20260621 06:52:22.970011 26381 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:22.972837 26388 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa2d5fbaf93948caaf5f57b3f1b1c6c0. Latest consensus state: current_term: 1 leader_uuid: "fa2d5fbaf93948caaf5f57b3f1b1c6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2d5fbaf93948caaf5f57b3f1b1c6c0" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37575 } } }
I20260621 06:52:22.972807 26387 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa2d5fbaf93948caaf5f57b3f1b1c6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2d5fbaf93948caaf5f57b3f1b1c6c0" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37575 } } }
I20260621 06:52:22.973277 26388 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:22.973282 26387 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:22.974895 26392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:22.979059 26392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:22.986732 26392 catalog_manager.cc:1347] Generated new cluster ID: 0643ceb4dd5f4b41b844735afb58efb7
I20260621 06:52:22.986892 26392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:22.999693 26392 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:23.001464 26392 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:23.022007 26392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa2d5fbaf93948caaf5f57b3f1b1c6c0: Generated new TSK 0
I20260621 06:52:23.022907 26392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:52:23.055585 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:23.047083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.20.120.254:37575)
0621 06:52:23.047286 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:23.047292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:23.047642 (+ 350us) server_negotiation.cc:407] Connection header received
0621 06:52:23.047855 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:23.047861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:23.047950 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:23.048119 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0621 06:52:23.048580 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:23.049742 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:23.051127 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:23.051855 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:23.051989 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:23.052455 (+ 466us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 06:52:23.054922 (+ 2467us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36143/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 06:52:23.055252 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46576: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36143/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":71}
W20260621 06:52:23.055709 26384 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46576: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36143/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260621 06:52:23.058197 26410 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:23.046928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:37575 (local address 127.0.0.1:46576)
0621 06:52:23.047539 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:23.047559 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:23.047678 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:23.048130 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:23.048134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:23.048142 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0621 06:52:23.048441 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:23.048449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:23.049883 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:23.049886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:23.050997 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:23.051007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:23.052046 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:23.052050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:23.052165 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:23.052196 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 06:52:23.055127 (+ 2931us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36143/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 06:52:23.057750 (+ 2623us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.120.254:37575: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36143/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":2305,"thread_start_us":339,"threads_started":1}
I20260621 06:52:23.059233 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 26315
I20260621 06:52:23.060442 26376 generic_service.cc:196] Checking for leaks (request via RPC)
E20260621 06:52:23.067987 26411 webserver.cc:533] Webserver: sslize failed:
E20260621 06:52:23.068111 26411 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 06:52:23.149466 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 26315
2026-06-21T06:52:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (711 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-21T06:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:23Z Disabled control of system clock
I20260621 06:52:23.228888 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:42935
--webserver_interface=127.20.120.254
--webserver_port=0
--builtin_ntp_servers=127.20.120.212:42429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:42935
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/testchainca.cert with env {}
W20260621 06:52:23.542749 26416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:23.543021 26416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:23.543192 26416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:23.550100 26416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:23.550192 26416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:23.550254 26416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:23.550374 26416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:23.550464 26416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 06:52:23.550570 26416 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 06:52:23.550645 26416 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260621 06:52:23.560140 26416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:42429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:42935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:42935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:23.561414 26416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:23.562654 26416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:23.573948 26421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:23.573974 26424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:23.573881 26422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:23.575078 26416 server_base.cc:1034] running on GCE node
I20260621 06:52:23.575858 26416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:23.577157 26416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:23.578382 26416 hybrid_clock.cc:648] HybridClock initialized: now 1782024743578346 us; error 47 us; skew 500 ppm
I20260621 06:52:23.578724 26416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:52:23.580924 26416 webserver.cc:460] Webserver started at http://127.20.120.254:34133/ using document root <none> and password file <none>
I20260621 06:52:23.581619 26416 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:23.581749 26416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:23.582104 26416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:23.584873 26416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "56f5268d437b44edbe15cca0fa865b0d"
format_stamp: "Formatted at 2026-06-21 06:52:23 on dist-test-slave-gmf2"
server_key: "35eb048b2a414dae29cef3e09b808e3f"
server_key_iv: "ff80056cd54b5e565438d9c7c9521b6a"
server_key_version: "encryptionkey@0"
I20260621 06:52:23.585628 26416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "56f5268d437b44edbe15cca0fa865b0d"
format_stamp: "Formatted at 2026-06-21 06:52:23 on dist-test-slave-gmf2"
server_key: "35eb048b2a414dae29cef3e09b808e3f"
server_key_iv: "ff80056cd54b5e565438d9c7c9521b6a"
server_key_version: "encryptionkey@0"
I20260621 06:52:23.590456 26416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260621 06:52:23.593482 26430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:23.594861 26416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:23.595024 26416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "56f5268d437b44edbe15cca0fa865b0d"
format_stamp: "Formatted at 2026-06-21 06:52:23 on dist-test-slave-gmf2"
server_key: "35eb048b2a414dae29cef3e09b808e3f"
server_key_iv: "ff80056cd54b5e565438d9c7c9521b6a"
server_key_version: "encryptionkey@0"
I20260621 06:52:23.595217 26416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:23.611593 26416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:23.612387 26416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:23.612670 26416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:23.630412 26416 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:42935
I20260621 06:52:23.630432 26481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:42935 every 8 connection(s)
I20260621 06:52:23.631706 26416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:23.635278 26482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:23.636345 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 26416
I20260621 06:52:23.636528 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:23.637176 20963 external_mini_cluster.cc:1428] Setting key 1fc12ea1006b678403e4d9cab1aaa415
I20260621 06:52:23.649354 26482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d: Bootstrap starting.
I20260621 06:52:23.651400 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:23.639044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.20.120.254:42935)
0621 06:52:23.640097 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:23.640112 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:23.640160 (+ 48us) server_negotiation.cc:407] Connection header received
0621 06:52:23.641758 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:23.641785 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:23.642418 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:23.642865 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:23.643381 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:23.645159 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:23.646421 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:23.647490 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:23.647648 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:23.648732 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:23.648743 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:23.648761 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:52:23.649456 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:23.649543 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:52:23.649566 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:23.649676 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:23.649867 (+ 191us) server_negotiation.cc:299] Negotiation successful
0621 06:52:23.650196 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":309,"threads_started":1}
I20260621 06:52:23.652877 26482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:23.653971 26482 log.cc:824] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:23.655165 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:52:23.657135 26482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d: No bootstrap required, opened a new log
I20260621 06:52:23.664608 26482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f5268d437b44edbe15cca0fa865b0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42935 } }
I20260621 06:52:23.664951 26482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:23.665026 26482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56f5268d437b44edbe15cca0fa865b0d, State: Initialized, Role: FOLLOWER
I20260621 06:52:23.665486 26482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f5268d437b44edbe15cca0fa865b0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42935 } }
I20260621 06:52:23.665674 26482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:23.665764 26482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:23.665880 26482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:23.667984 26482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f5268d437b44edbe15cca0fa865b0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42935 } }
I20260621 06:52:23.668570 26482 leader_election.cc:302] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56f5268d437b44edbe15cca0fa865b0d; no voters:
I20260621 06:52:23.669067 26482 leader_election.cc:288] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:23.669203 26487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:23.669874 26487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [term 1 LEADER]: Becoming Leader. State: Replica: 56f5268d437b44edbe15cca0fa865b0d, State: Running, Role: LEADER
I20260621 06:52:23.670377 26487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f5268d437b44edbe15cca0fa865b0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42935 } }
I20260621 06:52:23.671094 26482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:23.673302 26489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56f5268d437b44edbe15cca0fa865b0d. Latest consensus state: current_term: 1 leader_uuid: "56f5268d437b44edbe15cca0fa865b0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f5268d437b44edbe15cca0fa865b0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42935 } } }
I20260621 06:52:23.673714 26489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:23.673556 26488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56f5268d437b44edbe15cca0fa865b0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f5268d437b44edbe15cca0fa865b0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42935 } } }
I20260621 06:52:23.675474 26488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:23.676229 26497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:23.680416 26497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:23.688599 26497 catalog_manager.cc:1347] Generated new cluster ID: ac4ea5e342a140a2b8e2bce79e17f8c1
I20260621 06:52:23.688714 26497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:23.719987 26497 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:23.721254 26497 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:23.730888 26497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56f5268d437b44edbe15cca0fa865b0d: Generated new TSK 0
I20260621 06:52:23.731626 26497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:52:23.821048 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:23.750095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.20.120.254:42935)
0621 06:52:23.750331 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:23.750337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:23.750671 (+ 334us) server_negotiation.cc:407] Connection header received
0621 06:52:23.750881 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:23.750885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:23.750949 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:23.751063 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0621 06:52:23.751589 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:23.752325 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:23.754322 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:23.754766 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:23.754853 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:23.755384 (+ 531us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 06:52:23.820376 (+ 64992us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 06:52:23.820751 (+ 375us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48232: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20260621 06:52:23.821175 26485 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48232: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260621 06:52:23.821183 26511 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:23.749997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:42935 (local address 127.0.0.1:48232)
0621 06:52:23.750573 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:23.750592 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:23.750731 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:23.751106 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:23.751111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:23.751124 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0621 06:52:23.751427 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:23.751439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:23.752471 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:23.752477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:23.754197 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:23.754211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:23.754925 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:23.754932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:23.755106 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:23.755147 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 06:52:23.820654 (+ 65507us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 06:52:23.820888 (+ 234us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.120.254:42935: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":305,"threads_started":1}
I20260621 06:52:23.822345 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 26416
I20260621 06:52:23.823429 26477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:23.920394 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 26416
2026-06-21T06:52:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (750 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (395 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (399 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:52:24 dist-test-slave-gmf2 krb5kdc[26526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:52:24 dist-test-slave-gmf2 krb5kdc[26526](info): set up 2 sockets
Jun 21 06:52:24 dist-test-slave-gmf2 krb5kdc[26526](info): commencing operation
krb5kdc: starting...
W20260621 06:52:25.810629 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:52:25 dist-test-slave-gmf2 krb5kdc[26526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:52:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:52:26.013310 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:36257
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:36257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:26.325522 26539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:26.325816 26539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:26.326005 26539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:26.333063 26539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:26.333175 26539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:26.333294 26539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:26.333400 26539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:26.333484 26539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:26.342895 26539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:36257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:36257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:26.344250 26539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:26.345532 26539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:26.357115 26544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:26.357152 26545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:26.357270 26547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:26.357868 26539 server_base.cc:1034] running on GCE node
I20260621 06:52:26.358687 26539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:26.360178 26539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:26.361413 26539 hybrid_clock.cc:648] HybridClock initialized: now 1782024746361341 us; error 52 us; skew 500 ppm
I20260621 06:52:26.361814 26539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:26 dist-test-slave-gmf2 krb5kdc[26526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024746, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:26.364895 26539 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:52:26.366252 26539 webserver.cc:460] Webserver started at http://127.20.120.254:45875/ using document root <none> and password file <none>
I20260621 06:52:26.366871 26539 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:26.366964 26539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:26.367230 26539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:26.370163 26539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "5a6bf4f3c57f4492b1005b5f8ce83720"
format_stamp: "Formatted at 2026-06-21 06:52:26 on dist-test-slave-gmf2"
server_key: "74e107a0c4bfbb53e2b4764f98f85afc"
server_key_iv: "058f51c2f2efb5b24bacd85a444c9d9e"
server_key_version: "encryptionkey@0"
I20260621 06:52:26.370882 26539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "5a6bf4f3c57f4492b1005b5f8ce83720"
format_stamp: "Formatted at 2026-06-21 06:52:26 on dist-test-slave-gmf2"
server_key: "74e107a0c4bfbb53e2b4764f98f85afc"
server_key_iv: "058f51c2f2efb5b24bacd85a444c9d9e"
server_key_version: "encryptionkey@0"
I20260621 06:52:26.375768 26539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:52:26.379099 26554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:26.380420 26539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:52:26.380597 26539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "5a6bf4f3c57f4492b1005b5f8ce83720"
format_stamp: "Formatted at 2026-06-21 06:52:26 on dist-test-slave-gmf2"
server_key: "74e107a0c4bfbb53e2b4764f98f85afc"
server_key_iv: "058f51c2f2efb5b24bacd85a444c9d9e"
server_key_version: "encryptionkey@0"
I20260621 06:52:26.380764 26539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:26.403860 26539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:26.408278 26539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:26.408579 26539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:26.426973 26539 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:36257
I20260621 06:52:26.427009 26605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:36257 every 8 connection(s)
I20260621 06:52:26.428354 26539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:26.431981 26606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:26.433162 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 26539
I20260621 06:52:26.433372 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:26.434090 20963 external_mini_cluster.cc:1428] Setting key 5ecb2d8aee959179c89e5c65b2d270d6
I20260621 06:52:26.446353 26606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720: Bootstrap starting.
Jun 21 06:52:26 dist-test-slave-gmf2 krb5kdc[26526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:26.449563 26606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:26.450920 26606 log.cc:824] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:26.453508 26606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720: No bootstrap required, opened a new log
I20260621 06:52:26.462528 26609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:26.436123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.20.120.254:36257)
0621 06:52:26.436931 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:26.436950 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:26.437013 (+ 63us) server_negotiation.cc:407] Connection header received
0621 06:52:26.438808 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:26.438838 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:26.439396 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:26.439867 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:26.441140 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:26.443202 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:26.444417 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:26.445505 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:26.445684 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:26.449776 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:26.449788 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:26.449824 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:26.449875 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:26.458147 (+ 8272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:26.458813 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:26.458822 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:26.458830 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:26.459038 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:26.459367 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:26.459373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:26.459376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:26.460057 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:26.460367 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:26.460716 (+ 349us) server_negotiation.cc:299] Negotiation successful
0621 06:52:26.461158 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":335,"threads_started":1}
I20260621 06:52:26.463488 26606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6bf4f3c57f4492b1005b5f8ce83720" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36257 } }
I20260621 06:52:26.463819 26606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:26.463935 26606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6bf4f3c57f4492b1005b5f8ce83720, State: Initialized, Role: FOLLOWER
I20260621 06:52:26.464542 26606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6bf4f3c57f4492b1005b5f8ce83720" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36257 } }
I20260621 06:52:26.464768 26606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:26.464903 26606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:26.465107 26606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:26.467366 26606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6bf4f3c57f4492b1005b5f8ce83720" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36257 } }
I20260621 06:52:26.467846 26606 leader_election.cc:302] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a6bf4f3c57f4492b1005b5f8ce83720; no voters:
I20260621 06:52:26.468379 26606 leader_election.cc:288] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:26.468509 26611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:26.469213 26611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [term 1 LEADER]: Becoming Leader. State: Replica: 5a6bf4f3c57f4492b1005b5f8ce83720, State: Running, Role: LEADER
I20260621 06:52:26.469738 26611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6bf4f3c57f4492b1005b5f8ce83720" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36257 } }
I20260621 06:52:26.470378 26606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:26.472491 26612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a6bf4f3c57f4492b1005b5f8ce83720" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6bf4f3c57f4492b1005b5f8ce83720" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36257 } } }
I20260621 06:52:26.472360 26613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6bf4f3c57f4492b1005b5f8ce83720. Latest consensus state: current_term: 1 leader_uuid: "5a6bf4f3c57f4492b1005b5f8ce83720" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6bf4f3c57f4492b1005b5f8ce83720" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36257 } } }
I20260621 06:52:26.472910 26613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:26.473621 26612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:26.474731 26618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:26.479930 26618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:26.488984 26618 catalog_manager.cc:1347] Generated new cluster ID: c22063ff994746f8b3dfd054f42de9d2
I20260621 06:52:26.489111 26618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:26.500399 26618 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:26.501649 26618 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:26.506677 26618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a6bf4f3c57f4492b1005b5f8ce83720: Generated new TSK 0
I20260621 06:52:26.507663 26618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:52:26.556772 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:36257
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:26.885270 26632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:26.885582 26632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:26.885739 26632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:26.893196 26632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:26.893350 26632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:26.893597 26632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:52:26.903645 26632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:36257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:26.904974 26632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:26.906373 26632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:26.919623 26637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:26.919665 26640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:26.919687 26638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:26.920953 26632 server_base.cc:1034] running on GCE node
I20260621 06:52:26.921542 26632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:26.922518 26632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:26.923705 26632 hybrid_clock.cc:648] HybridClock initialized: now 1782024746923666 us; error 35 us; skew 500 ppm
I20260621 06:52:26.924003 26632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:26 dist-test-slave-gmf2 krb5kdc[26526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024746, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:26.927142 26632 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:52:26.928577 26632 webserver.cc:460] Webserver started at http://127.20.120.193:38299/ using document root <none> and password file <none>
I20260621 06:52:26.929199 26632 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:26.929303 26632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:26.929548 26632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:26.932427 26632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "d464d02192344350b02c0204e4a991b7"
format_stamp: "Formatted at 2026-06-21 06:52:26 on dist-test-slave-gmf2"
server_key: "284e485e003027f2a01b8195c0d88ad7"
server_key_iv: "4352fc549919b28c3e5bfd4d85b96b6e"
server_key_version: "encryptionkey@0"
I20260621 06:52:26.933183 26632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "d464d02192344350b02c0204e4a991b7"
format_stamp: "Formatted at 2026-06-21 06:52:26 on dist-test-slave-gmf2"
server_key: "284e485e003027f2a01b8195c0d88ad7"
server_key_iv: "4352fc549919b28c3e5bfd4d85b96b6e"
server_key_version: "encryptionkey@0"
I20260621 06:52:26.938208 26632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260621 06:52:26.941331 26647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:26.942663 26632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:26.942842 26632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "d464d02192344350b02c0204e4a991b7"
format_stamp: "Formatted at 2026-06-21 06:52:26 on dist-test-slave-gmf2"
server_key: "284e485e003027f2a01b8195c0d88ad7"
server_key_iv: "4352fc549919b28c3e5bfd4d85b96b6e"
server_key_version: "encryptionkey@0"
I20260621 06:52:26.943054 26632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:26.962800 26632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:26.967082 26632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:26.967410 26632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:26.968400 26632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:26.969811 26632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:26.969920 26632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:26.970104 26632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:26.970208 26632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:27.005801 26632 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:36739
I20260621 06:52:27.005939 26759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:36739 every 8 connection(s)
I20260621 06:52:27.007380 26632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:52:27.009034 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 26632
I20260621 06:52:27.009244 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:52:27.009968 20963 external_mini_cluster.cc:1428] Setting key 026462742a1a0dd88a31abbfeaf2a0fd
Jun 21 06:52:27 dist-test-slave-gmf2 krb5kdc[26526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024746, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:27.029429 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:27.010057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:39515 (local address 127.20.120.254:36257)
0621 06:52:27.010895 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:27.010900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:27.013964 (+ 3064us) server_negotiation.cc:407] Connection header received
0621 06:52:27.014610 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:27.014615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:27.014711 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:27.014822 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:27.016769 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:27.017508 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:27.019103 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:27.019509 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:27.019650 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:27.023729 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:27.023734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:27.023737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:27.023769 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:27.026417 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:27.027141 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:27.027147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:27.027158 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:27.027271 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:27.027705 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:27.027710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:27.027713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:27.027979 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:27.028141 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:27.028849 (+ 708us) server_negotiation.cc:299] Negotiation successful
0621 06:52:27.029071 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":573,"threads_started":1}
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
I20260621 06:52:27.030568 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:27.010769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:36257 (local address 127.20.120.193:39515)
0621 06:52:27.011879 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:27.011925 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:27.014327 (+ 2402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:27.015204 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:27.015216 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:27.015979 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:27.016574 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:27.016590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:27.017668 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:27.017673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:27.018965 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:27.018976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:27.019745 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:27.019754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:27.019978 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:27.020647 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:27.020672 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:27.023498 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:27.026565 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:27.026573 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:27.026591 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:27.027012 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:27.027387 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:27.027392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:27.027396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:27.027585 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:27.028106 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:27.028114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:27.028467 (+ 353us) client_negotiation.cc:769] Sending connection context
0621 06:52:27.028741 (+ 274us) client_negotiation.cc:240] Negotiation successful
0621 06:52:27.029272 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":418,"threads_started":1}
I20260621 06:52:27.032147 26760 heartbeater.cc:347] Connected to a master server at 127.20.120.254:36257
I20260621 06:52:27.032492 26760 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
I20260621 06:52:27.033300 26760 heartbeater.cc:511] Master 127.20.120.254:36257 requested a full tablet report, sending...
I20260621 06:52:27.035506 26571 ts_manager.cc:194] Registered new tserver with Master: d464d02192344350b02c0204e4a991b7 (127.20.120.193:36739)
I20260621 06:52:27.036947 26571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:39515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:52:27.057781 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:36257
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:27.400403 26766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:27.400713 26766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:27.400888 26766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:27.408609 26766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:27.408738 26766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:27.408946 26766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:52:27.419301 26766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:36257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:27.420727 26766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:27.422228 26766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:27.436761 26774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:27.436663 26772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:27.438341 26771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:27.440030 26766 server_base.cc:1034] running on GCE node
I20260621 06:52:27.440881 26766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:27.442582 26766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:27.443851 26766 hybrid_clock.cc:648] HybridClock initialized: now 1782024747443777 us; error 62 us; skew 500 ppm
I20260621 06:52:27.444274 26766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:27 dist-test-slave-gmf2 krb5kdc[26526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024747, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:27.448057 26766 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:52:27.449602 26766 webserver.cc:460] Webserver started at http://127.20.120.194:32773/ using document root <none> and password file <none>
I20260621 06:52:27.450485 26766 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:27.450628 26766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:27.450956 26766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:27.453876 26766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "c321c11fb2c94bf598954708bea55635"
format_stamp: "Formatted at 2026-06-21 06:52:27 on dist-test-slave-gmf2"
server_key: "adc974fd76569274c23c4cd091ec520e"
server_key_iv: "85c728fe9ff52d6b47751598e92ddd31"
server_key_version: "encryptionkey@0"
I20260621 06:52:27.454784 26766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "c321c11fb2c94bf598954708bea55635"
format_stamp: "Formatted at 2026-06-21 06:52:27 on dist-test-slave-gmf2"
server_key: "adc974fd76569274c23c4cd091ec520e"
server_key_iv: "85c728fe9ff52d6b47751598e92ddd31"
server_key_version: "encryptionkey@0"
I20260621 06:52:27.459851 26766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:52:27.463383 26781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:27.464809 26766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260621 06:52:27.464977 26766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "c321c11fb2c94bf598954708bea55635"
format_stamp: "Formatted at 2026-06-21 06:52:27 on dist-test-slave-gmf2"
server_key: "adc974fd76569274c23c4cd091ec520e"
server_key_iv: "85c728fe9ff52d6b47751598e92ddd31"
server_key_version: "encryptionkey@0"
I20260621 06:52:27.465189 26766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:27.496343 26766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:27.500574 26766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:27.500874 26766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:27.501863 26766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:27.503211 26766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:27.503281 26766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:27.503367 26766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:27.503410 26766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:27.535749 26766 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:41817
I20260621 06:52:27.535791 26893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:41817 every 8 connection(s)
I20260621 06:52:27.537074 26766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:52:27.539055 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 26766
I20260621 06:52:27.539224 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:52:27.539757 20963 external_mini_cluster.cc:1428] Setting key 87e35ed75c7cb85ee81666fabbc67824
Jun 21 06:52:27 dist-test-slave-gmf2 krb5kdc[26526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024747, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:27.559195 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:27.539729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:35911 (local address 127.20.120.254:36257)
0621 06:52:27.540608 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:27.540613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:27.541544 (+ 931us) server_negotiation.cc:407] Connection header received
0621 06:52:27.543479 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:27.543482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:27.543543 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:27.543624 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:27.545817 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:27.546597 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:27.548632 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:27.549056 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:27.549161 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:27.552965 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:27.552970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:27.552974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:27.553006 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:27.555664 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:27.556658 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:27.556662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:27.556664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:27.556795 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:27.557324 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:27.557327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:27.557329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:27.557584 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:27.557689 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:27.558774 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0621 06:52:27.558944 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":634,"threads_started":1}
I20260621 06:52:27.560850 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:27.540488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:36257 (local address 127.20.120.194:35911)
0621 06:52:27.541377 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:27.541424 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:27.543124 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:27.543933 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:27.543947 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:27.544780 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:27.545624 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:27.545652 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:27.546773 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:27.546781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:27.548471 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:27.548483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:27.549250 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:27.549255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:27.549462 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:27.550237 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:27.550267 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:27.552721 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:27.555819 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:27.555832 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:27.555862 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:27.556534 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:27.556917 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:27.556923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:27.556928 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:27.557203 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:27.557722 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:27.557734 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:27.558319 (+ 585us) client_negotiation.cc:769] Sending connection context
0621 06:52:27.558679 (+ 360us) client_negotiation.cc:240] Negotiation successful
0621 06:52:27.559292 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":390,"threads_started":1}
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
I20260621 06:52:27.562369 26894 heartbeater.cc:347] Connected to a master server at 127.20.120.254:36257
I20260621 06:52:27.562631 26894 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
I20260621 06:52:27.563472 26894 heartbeater.cc:511] Master 127.20.120.254:36257 requested a full tablet report, sending...
I20260621 06:52:27.565192 26571 ts_manager.cc:194] Registered new tserver with Master: c321c11fb2c94bf598954708bea55635 (127.20.120.194:41817)
I20260621 06:52:27.566443 26571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:35911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:52:27.587504 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:36257
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:27.909492 26900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:27.909819 26900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:27.910032 26900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:27.917465 26900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:27.917598 26900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:27.917784 26900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:52:27.927234 26900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:36257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:27.928531 26900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:27.929823 26900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:27.942891 26905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:27.942974 26908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:27.942821 26906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:27.943819 26900 server_base.cc:1034] running on GCE node
I20260621 06:52:27.944337 26900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:27.945166 26900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:27.946375 26900 hybrid_clock.cc:648] HybridClock initialized: now 1782024747946308 us; error 33 us; skew 500 ppm
I20260621 06:52:27.946728 26900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:27 dist-test-slave-gmf2 krb5kdc[26526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024747, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:27.949767 26900 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:52:27.951097 26900 webserver.cc:460] Webserver started at http://127.20.120.195:39349/ using document root <none> and password file <none>
I20260621 06:52:27.951776 26900 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:27.951941 26900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:27.952242 26900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:27.955139 26900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "1dadc7b3be784bd48350f68aa86abda2"
format_stamp: "Formatted at 2026-06-21 06:52:27 on dist-test-slave-gmf2"
server_key: "fc2bf13390d543c4d0133cc6a378d18d"
server_key_iv: "71af0297ee0a4906393ad7ccf5a9e09e"
server_key_version: "encryptionkey@0"
I20260621 06:52:27.955885 26900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "1dadc7b3be784bd48350f68aa86abda2"
format_stamp: "Formatted at 2026-06-21 06:52:27 on dist-test-slave-gmf2"
server_key: "fc2bf13390d543c4d0133cc6a378d18d"
server_key_iv: "71af0297ee0a4906393ad7ccf5a9e09e"
server_key_version: "encryptionkey@0"
I20260621 06:52:27.960659 26900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 06:52:27.963798 26915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:27.965138 26900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:52:27.965297 26900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "1dadc7b3be784bd48350f68aa86abda2"
format_stamp: "Formatted at 2026-06-21 06:52:27 on dist-test-slave-gmf2"
server_key: "fc2bf13390d543c4d0133cc6a378d18d"
server_key_iv: "71af0297ee0a4906393ad7ccf5a9e09e"
server_key_version: "encryptionkey@0"
I20260621 06:52:27.965502 26900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:27.979331 26900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:27.983085 26900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:27.983407 26900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:27.984367 26900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:27.985643 26900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:27.985741 26900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:27.985883 26900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:27.985997 26900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260621 06:52:28.019488 26900 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:43597
I20260621 06:52:28.019573 27027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:43597 every 8 connection(s)
I20260621 06:52:28.020757 26900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:52:28.027297 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 26900
I20260621 06:52:28.027485 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:52:28.028010 20963 external_mini_cluster.cc:1428] Setting key d601db19baff69eefa3916ec8952fba7
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024747, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:28.039953 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:28.023302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:47337 (local address 127.20.120.254:36257)
0621 06:52:28.023524 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:28.023531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:28.024659 (+ 1128us) server_negotiation.cc:407] Connection header received
0621 06:52:28.026403 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:28.026408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:28.026499 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:28.026588 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:28.028597 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.029521 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.031452 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.031814 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.031918 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:28.035086 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:28.035090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:28.035093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:28.035122 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:28.037397 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.038085 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.038090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.038092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.038204 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.038560 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.038563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.038565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.038806 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:28.038926 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:28.039563 (+ 637us) server_negotiation.cc:299] Negotiation successful
0621 06:52:28.039723 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 06:52:28.040551 26760 heartbeater.cc:503] Master 127.20.120.254:36257 was elected leader, sending a full tablet report...
I20260621 06:52:28.041049 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:28.023783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:36257 (local address 127.20.120.195:47337)
0621 06:52:28.024494 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:28.024530 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:28.026136 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:28.026785 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:28.026793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:28.027576 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:28.028395 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:28.028420 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.029673 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.029679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:28.031334 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:28.031345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.032003 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.032008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:28.032194 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:28.032794 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:28.032815 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:28.034917 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:28.037511 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:28.037517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:28.037529 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:28.037980 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:28.038296 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:28.038299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:28.038301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:28.038468 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:28.038910 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:28.038915 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:28.039206 (+ 291us) client_negotiation.cc:769] Sending connection context
0621 06:52:28.039452 (+ 246us) client_negotiation.cc:240] Negotiation successful
0621 06:52:28.039841 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":286,"threads_started":1}
I20260621 06:52:28.042794 27028 heartbeater.cc:347] Connected to a master server at 127.20.120.254:36257
I20260621 06:52:28.043092 27028 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:28.043759 27028 heartbeater.cc:511] Master 127.20.120.254:36257 requested a full tablet report, sending...
I20260621 06:52:28.045034 26571 ts_manager.cc:194] Registered new tserver with Master: 1dadc7b3be784bd48350f68aa86abda2 (127.20.120.195:43597)
I20260621 06:52:28.045763 26571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:47337
I20260621 06:52:28.055546 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/test-admin.keytab.
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:28.081316 20963 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260621 06:52:28.086333 27035 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:28.083974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.20.120.193:36739)
0621 06:52:28.084669 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:28.084676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:28.084704 (+ 28us) server_negotiation.cc:407] Connection header received
0621 06:52:28.084765 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:28.084771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:28.084984 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:28.085113 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:28.085982 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35106: BlockingRecv error: recv got EOF from 127.0.0.1:35106 (error 108)
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":352,"threads_started":1}
W20260621 06:52:28.086441 20963 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5cc' of type FILE
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:28.087971 20963 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:28.089265 20963 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:52:28.103570 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:28.090806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.20.120.193:36739)
0621 06:52:28.091006 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:28.091016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:28.091270 (+ 254us) server_negotiation.cc:407] Connection header received
0621 06:52:28.091408 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:28.091412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:28.091466 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:28.091543 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:28.092449 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.093380 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.094607 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.095192 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.095285 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:28.098642 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:28.098652 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:28.098667 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:28.098697 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:28.101174 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.101724 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.101731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.101733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.101838 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.102226 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.102229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.102231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.102556 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:28.102770 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:28.103026 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 06:52:28.103279 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260621 06:52:28.109146 27035 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:28.107440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35122 (local address 127.20.120.193:36739)
0621 06:52:28.107632 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:28.107640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:28.107904 (+ 264us) server_negotiation.cc:407] Connection header received
0621 06:52:28.108087 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:28.108091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:28.108141 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:28.108236 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:28.108903 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35122: BlockingRecv error: recv got EOF from 127.0.0.1:35122 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 06:52:28.109599 20963 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5cc)
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:28.111064 20963 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:28.112313 20963 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:52:28.125319 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:28.113423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.20.120.193:36739)
0621 06:52:28.113628 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:28.113635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:28.113901 (+ 266us) server_negotiation.cc:407] Connection header received
0621 06:52:28.114121 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:28.114126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:28.114185 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:28.114279 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:28.115189 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.115970 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.117069 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.117498 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.117591 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:28.120713 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:28.120717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:28.120720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:28.120748 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:28.123231 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.123825 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.123828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.123830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.123934 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.124220 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.124223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.124224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.124448 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:28.124657 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:28.124871 (+ 214us) server_negotiation.cc:299] Negotiation successful
0621 06:52:28.125030 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260621 06:52:28.129209 27035 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:28.127615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.20.120.193:36739)
0621 06:52:28.127832 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:28.127845 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:28.128055 (+ 210us) server_negotiation.cc:407] Connection header received
0621 06:52:28.128225 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:28.128230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:28.128304 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:28.128398 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:28.128992 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35150: BlockingRecv error: recv got EOF from 127.0.0.1:35150 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260621 06:52:28.129406 20963 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782024700637160-20963-0/krb5kdc/krb5cc)
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:28.130829 20963 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:28.132025 20963 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:52:28.145548 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:28.133177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.20.120.193:36739)
0621 06:52:28.133350 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:28.133356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:28.133683 (+ 327us) server_negotiation.cc:407] Connection header received
0621 06:52:28.133852 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:28.133864 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:28.134007 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:28.134102 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:28.135018 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.135726 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.136946 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.137350 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.137480 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:28.140836 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:28.140842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:28.140854 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:28.140889 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:28.143177 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.143762 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.143767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.143771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.143903 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.144270 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.144276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.144279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.144537 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:28.144837 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:28.145059 (+ 222us) server_negotiation.cc:299] Negotiation successful
0621 06:52:28.145253 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 06:52:28.147127 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 26632
I20260621 06:52:28.158130 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:28.147780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.20.120.193:36739)
0621 06:52:28.147937 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:28.147941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:28.148395 (+ 454us) server_negotiation.cc:407] Connection header received
0621 06:52:28.148481 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:28.148484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:28.148541 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:28.148614 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:28.149464 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.150224 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.151417 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.151770 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.151856 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:28.153734 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:28.153738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:28.153741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:28.153764 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:28.156060 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.156578 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.156581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.156583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.156701 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.157013 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.157016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.157018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.157238 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:28.157422 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:28.157643 (+ 221us) server_negotiation.cc:299] Negotiation successful
0621 06:52:28.157791 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 06:52:28.158672 26755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:28.240015 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 26632
I20260621 06:52:28.256203 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 26766
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:52:28.271085 27048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:28.257035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.20.120.194:41817)
0621 06:52:28.257549 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:28.257556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:28.257582 (+ 26us) server_negotiation.cc:407] Connection header received
0621 06:52:28.257641 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:28.257647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:28.257873 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:28.258063 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:28.258914 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.259905 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.261036 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.261560 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.261649 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:28.265272 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:28.265282 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:28.265297 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:28.265327 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:28.268139 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.268817 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.268825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.268827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.268949 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.269325 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.269332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.269335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.269754 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:28.270095 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:28.270446 (+ 351us) server_negotiation.cc:299] Negotiation successful
0621 06:52:28.270722 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":262,"threads_started":1}
I20260621 06:52:28.272373 26889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:28.352463 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 26766
I20260621 06:52:28.367702 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 26900
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[26526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:52:28.383639 27050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:28.368710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.20.120.195:43597)
0621 06:52:28.369404 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:28.369411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:28.369456 (+ 45us) server_negotiation.cc:407] Connection header received
0621 06:52:28.369556 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:28.369563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:28.369782 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:28.369983 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:28.371231 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.372272 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.373483 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:28.374135 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:28.374232 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:28.378106 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:28.378117 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:28.378132 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:28.378163 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:28.380970 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.381639 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.381646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.381648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.381763 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:28.382167 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:28.382173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:28.382177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:28.382500 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:28.382804 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:28.383048 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 06:52:28.383293 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":375,"threads_started":1}
I20260621 06:52:28.384991 27023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:28.463944 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 26900
I20260621 06:52:28.479739 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 26539
I20260621 06:52:28.480872 26601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:28.579885 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 26539
2026-06-21T06:52:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3869 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[27055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[27055](info): set up 2 sockets
Jun 21 06:52:28 dist-test-slave-gmf2 krb5kdc[27055](info): commencing operation
krb5kdc: starting...
W20260621 06:52:29.662688 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:52:29 dist-test-slave-gmf2 krb5kdc[27055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:52:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.254.keytab.
Entry for principal oryx/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.254.keytab.
I20260621 06:52:29.865264 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:45135
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.254.keytab
--principal=oryx/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:45135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:30.190793 27068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:30.191095 27068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:30.191269 27068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:30.198571 27068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:30.198679 27068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:30.198783 27068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:30.198884 27068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:30.198963 27068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:30.208814 27068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:45135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.254.keytab
--principal=oryx/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:45135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:30.210206 27068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:30.211560 27068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:30.223320 27073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:30.223423 27074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:30.223403 27076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:30.224025 27068 server_base.cc:1034] running on GCE node
I20260621 06:52:30.224970 27068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:30.226356 27068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:30.227555 27068 hybrid_clock.cc:648] HybridClock initialized: now 1782024750227514 us; error 30 us; skew 500 ppm
I20260621 06:52:30.227907 27068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:30 dist-test-slave-gmf2 krb5kdc[27055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024750, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:30.231141 27068 init.cc:377] Logged in from keytab as oryx/127.20.120.254@KRBTEST.COM (short username oryx)
I20260621 06:52:30.232599 27068 webserver.cc:460] Webserver started at http://127.20.120.254:34485/ using document root <none> and password file <none>
I20260621 06:52:30.233289 27068 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:30.233421 27068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:30.233723 27068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:30.236788 27068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "85e295b70bac43c0a54c15c239835d0d"
format_stamp: "Formatted at 2026-06-21 06:52:30 on dist-test-slave-gmf2"
server_key: "e73a5a28a458c6463e9c25738cc1abd7"
server_key_iv: "4889bbd38d96dfb4ca9403d42b7c22d6"
server_key_version: "encryptionkey@0"
I20260621 06:52:30.237562 27068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "85e295b70bac43c0a54c15c239835d0d"
format_stamp: "Formatted at 2026-06-21 06:52:30 on dist-test-slave-gmf2"
server_key: "e73a5a28a458c6463e9c25738cc1abd7"
server_key_iv: "4889bbd38d96dfb4ca9403d42b7c22d6"
server_key_version: "encryptionkey@0"
I20260621 06:52:30.242640 27068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 06:52:30.245872 27083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:30.247253 27068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:52:30.247438 27068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "85e295b70bac43c0a54c15c239835d0d"
format_stamp: "Formatted at 2026-06-21 06:52:30 on dist-test-slave-gmf2"
server_key: "e73a5a28a458c6463e9c25738cc1abd7"
server_key_iv: "4889bbd38d96dfb4ca9403d42b7c22d6"
server_key_version: "encryptionkey@0"
I20260621 06:52:30.247632 27068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:30.271703 27068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:30.276291 27068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:30.276592 27068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:30.295758 27068 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:45135
I20260621 06:52:30.295781 27134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:45135 every 8 connection(s)
I20260621 06:52:30.297151 27068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:30.300755 27135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:30.305980 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 27068
I20260621 06:52:30.306175 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:30.306954 20963 external_mini_cluster.cc:1428] Setting key cd1070028e72ec6c14b60f59a6eb81fd
I20260621 06:52:30.311906 27135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d: Bootstrap starting.
I20260621 06:52:30.314996 27135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:30.315975 27135 log.cc:824] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:30.318904 27135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d: No bootstrap required, opened a new log
Jun 21 06:52:30 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:30.327536 27135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } }
I20260621 06:52:30.328032 27135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:30.328140 27135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e295b70bac43c0a54c15c239835d0d, State: Initialized, Role: FOLLOWER
I20260621 06:52:30.328814 27135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } }
I20260621 06:52:30.329044 27135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:30.329185 27135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:30.329370 27135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:30.329481 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:30.309143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.20.120.254:45135)
0621 06:52:30.310021 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:30.310035 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:30.310075 (+ 40us) server_negotiation.cc:407] Connection header received
0621 06:52:30.311627 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:30.311649 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:30.312127 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:30.312511 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:30.313877 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:30.315332 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:30.316706 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:30.317540 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:30.317717 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:30.322170 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:30.322177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:30.322194 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:30.322226 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:30.325237 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:30.326029 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:30.326035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:30.326041 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:30.326210 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:30.326607 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:30.326610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:30.326612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:30.327144 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:30.327363 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:30.327803 (+ 440us) server_negotiation.cc:299] Negotiation successful
0621 06:52:30.328331 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":367,"threads_started":1}
I20260621 06:52:30.331781 27135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } }
I20260621 06:52:30.332391 27135 leader_election.cc:302] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85e295b70bac43c0a54c15c239835d0d; no voters:
I20260621 06:52:30.332996 27135 leader_election.cc:288] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:30.333096 27140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:30.333803 27140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 1 LEADER]: Becoming Leader. State: Replica: 85e295b70bac43c0a54c15c239835d0d, State: Running, Role: LEADER
I20260621 06:52:30.334307 27140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } }
I20260621 06:52:30.335031 27135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:30.337232 27142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e295b70bac43c0a54c15c239835d0d. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } }
I20260621 06:52:30.337667 27142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:30.341019 27146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:30.340463 27141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } }
I20260621 06:52:30.341141 27141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:30.346330 27146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:30.355751 27146 catalog_manager.cc:1347] Generated new cluster ID: 2660fb547c4648959bd33249505d04a5
I20260621 06:52:30.355902 27146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:30.371328 27146 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:30.373243 27146 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:30.388584 27146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d: Generated new TSK 0
I20260621 06:52:30.389743 27146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab.
Entry for principal oryx/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab.
I20260621 06:52:30.439618 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45135
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab
--principal=oryx/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:30.788404 27161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:30.788851 27161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:30.789026 27161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:30.796495 27161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:30.796617 27161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:30.796803 27161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:52:30.806685 27161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab
--principal=oryx/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:30.808030 27161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:30.809391 27161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:30.822542 27166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:30.822516 27167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:30.822697 27169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:30.823380 27161 server_base.cc:1034] running on GCE node
I20260621 06:52:30.823944 27161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:30.824788 27161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:30.826016 27161 hybrid_clock.cc:648] HybridClock initialized: now 1782024750825949 us; error 53 us; skew 500 ppm
I20260621 06:52:30.826399 27161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:30 dist-test-slave-gmf2 krb5kdc[27055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024750, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:30.829640 27161 init.cc:377] Logged in from keytab as oryx/127.20.120.193@KRBTEST.COM (short username oryx)
I20260621 06:52:30.831944 27161 webserver.cc:460] Webserver started at http://127.20.120.193:42851/ using document root <none> and password file <none>
I20260621 06:52:30.832634 27161 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:30.832762 27161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:30.833045 27161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:30.836445 27161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "dea92d7a226f4d5a8be376c02695cd4a"
format_stamp: "Formatted at 2026-06-21 06:52:30 on dist-test-slave-gmf2"
server_key: "2befbcc7f47f4d2127a847a7491a0880"
server_key_iv: "e7369706f252d9ee3aca9aae64672103"
server_key_version: "encryptionkey@0"
I20260621 06:52:30.837242 27161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "dea92d7a226f4d5a8be376c02695cd4a"
format_stamp: "Formatted at 2026-06-21 06:52:30 on dist-test-slave-gmf2"
server_key: "2befbcc7f47f4d2127a847a7491a0880"
server_key_iv: "e7369706f252d9ee3aca9aae64672103"
server_key_version: "encryptionkey@0"
I20260621 06:52:30.842595 27161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 06:52:30.846029 27176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:30.847381 27161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:52:30.847558 27161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "dea92d7a226f4d5a8be376c02695cd4a"
format_stamp: "Formatted at 2026-06-21 06:52:30 on dist-test-slave-gmf2"
server_key: "2befbcc7f47f4d2127a847a7491a0880"
server_key_iv: "e7369706f252d9ee3aca9aae64672103"
server_key_version: "encryptionkey@0"
I20260621 06:52:30.847771 27161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:30.874927 27161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:30.878957 27161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:30.879256 27161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:30.880209 27161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:30.881563 27161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:30.881662 27161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:30.881800 27161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:30.881868 27161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:30.914395 27161 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:40259
I20260621 06:52:30.914446 27288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:40259 every 8 connection(s)
I20260621 06:52:30.915676 27161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:52:30.922243 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 27161
I20260621 06:52:30.922442 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:52:30.923272 20963 external_mini_cluster.cc:1428] Setting key 01c596edde55670b0d826d8d633022aa
Jun 21 06:52:30 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024750, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:30.938522 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:30.918170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:43493 (local address 127.20.120.254:45135)
0621 06:52:30.918586 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:30.918591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:30.919613 (+ 1022us) server_negotiation.cc:407] Connection header received
0621 06:52:30.921358 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:30.921363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:30.921451 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:30.921563 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:30.923660 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:30.924482 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:30.926776 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:30.927154 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:30.927254 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:30.931700 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:30.931705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:30.931708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:30.931739 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:30.935164 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:30.935893 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:30.935901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:30.935908 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:30.936052 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:30.936522 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:30.936530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:30.936536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:30.936905 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:30.937061 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:30.937752 (+ 691us) server_negotiation.cc:299] Negotiation successful
0621 06:52:30.938028 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":216,"threads_started":1}
I20260621 06:52:30.939216 27292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:30.918793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.193:43493)
0621 06:52:30.919463 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:30.919502 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:30.921103 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:30.921817 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:30.921829 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:30.922614 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:30.923455 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:30.923482 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:30.924642 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:30.924646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:30.926618 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:30.926642 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:30.927387 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:30.927396 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:30.927736 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:30.928481 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:30.928514 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:30.931462 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:30.935305 (+ 3843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:30.935312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:30.935327 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:30.935759 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:30.936154 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:30.936157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:30.936159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:30.936383 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:30.937016 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:30.937022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:30.937354 (+ 332us) client_negotiation.cc:769] Sending connection context
0621 06:52:30.937594 (+ 240us) client_negotiation.cc:240] Negotiation successful
0621 06:52:30.938061 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":252,"threads_started":1}
I20260621 06:52:30.940825 27289 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45135
I20260621 06:52:30.941146 27289 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:30.942004 27289 heartbeater.cc:511] Master 127.20.120.254:45135 requested a full tablet report, sending...
I20260621 06:52:30.944273 27100 ts_manager.cc:194] Registered new tserver with Master: dea92d7a226f4d5a8be376c02695cd4a (127.20.120.193:40259)
WARNING: no policy specified for oryx/127.20.120.194@KRBTEST.COM; defaulting to no policy
I20260621 06:52:30.945760 27100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:43493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab.
Entry for principal oryx/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab.
I20260621 06:52:30.973068 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45135
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab
--principal=oryx/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:31.301347 27295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:31.301637 27295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:31.301893 27295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:31.309454 27295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:31.309566 27295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:31.309739 27295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:52:31.319667 27295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab
--principal=oryx/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:31.321007 27295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:31.322451 27295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:31.335680 27300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:31.335745 27303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:31.335654 27301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:31.336838 27295 server_base.cc:1034] running on GCE node
I20260621 06:52:31.337538 27295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:31.338572 27295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:31.339787 27295 hybrid_clock.cc:648] HybridClock initialized: now 1782024751339730 us; error 54 us; skew 500 ppm
I20260621 06:52:31.340113 27295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:31 dist-test-slave-gmf2 krb5kdc[27055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024751, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:31.343372 27295 init.cc:377] Logged in from keytab as oryx/127.20.120.194@KRBTEST.COM (short username oryx)
I20260621 06:52:31.344841 27295 webserver.cc:460] Webserver started at http://127.20.120.194:40245/ using document root <none> and password file <none>
I20260621 06:52:31.345531 27295 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:31.345659 27295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:31.346033 27295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:31.349036 27295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "0b8f4bce11874dd39985773f5e4a5e49"
format_stamp: "Formatted at 2026-06-21 06:52:31 on dist-test-slave-gmf2"
server_key: "01f3ab007ded71eef63fa13e0e6dd2e0"
server_key_iv: "fd89af56dfd774cbdd4b657addcd536c"
server_key_version: "encryptionkey@0"
I20260621 06:52:31.349856 27295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "0b8f4bce11874dd39985773f5e4a5e49"
format_stamp: "Formatted at 2026-06-21 06:52:31 on dist-test-slave-gmf2"
server_key: "01f3ab007ded71eef63fa13e0e6dd2e0"
server_key_iv: "fd89af56dfd774cbdd4b657addcd536c"
server_key_version: "encryptionkey@0"
I20260621 06:52:31.355614 27295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:52:31.358989 27310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:31.360239 27295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:52:31.360420 27295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "0b8f4bce11874dd39985773f5e4a5e49"
format_stamp: "Formatted at 2026-06-21 06:52:31 on dist-test-slave-gmf2"
server_key: "01f3ab007ded71eef63fa13e0e6dd2e0"
server_key_iv: "fd89af56dfd774cbdd4b657addcd536c"
server_key_version: "encryptionkey@0"
I20260621 06:52:31.360625 27295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:31.393703 27295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:31.397568 27295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:31.397856 27295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:31.398837 27295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:31.400128 27295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:31.400226 27295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:31.400367 27295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:31.400436 27295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:31.431792 27295 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:43333
I20260621 06:52:31.431828 27422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:43333 every 8 connection(s)
I20260621 06:52:31.433141 27295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:52:31.442973 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 27295
I20260621 06:52:31.443152 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:52:31.443732 20963 external_mini_cluster.cc:1428] Setting key 2bd9812a57c75bc4dc158b142447f8ca
Jun 21 06:52:31 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024751, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:31.456616 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:31.436085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:55989 (local address 127.20.120.254:45135)
0621 06:52:31.436334 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:31.436340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:31.438061 (+ 1721us) server_negotiation.cc:407] Connection header received
0621 06:52:31.440689 (+ 2628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:31.440698 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:31.440801 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:31.440916 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:31.442884 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:31.444023 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:31.446028 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:31.446665 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:31.446808 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:31.450526 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:31.450532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:31.450535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:31.450565 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:31.453685 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:31.454429 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:31.454434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:31.454438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:31.454565 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:31.454971 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:31.454976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:31.454979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:31.455237 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:31.455398 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:31.456068 (+ 670us) server_negotiation.cc:299] Negotiation successful
0621 06:52:31.456281 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 06:52:31.457600 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:31.436736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.194:55989)
0621 06:52:31.437821 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:31.437920 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:31.440350 (+ 2430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:31.441192 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:31.441205 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:31.441757 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:31.442646 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:31.442675 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:31.444199 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:31.444204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:31.445843 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:31.445855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:31.446862 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:31.446866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:31.447082 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:31.447771 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:31.447798 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:31.450301 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:31.453814 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:31.453820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:31.453834 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:31.454304 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:31.454677 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:31.454682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:31.454686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:31.454861 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:31.455372 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:31.455380 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:31.455715 (+ 335us) client_negotiation.cc:769] Sending connection context
0621 06:52:31.455981 (+ 266us) client_negotiation.cc:240] Negotiation successful
0621 06:52:31.456511 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":367,"threads_started":1}
I20260621 06:52:31.459039 27423 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45135
I20260621 06:52:31.459343 27423 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:31.460096 27423 heartbeater.cc:511] Master 127.20.120.254:45135 requested a full tablet report, sending...
I20260621 06:52:31.461568 27100 ts_manager.cc:194] Registered new tserver with Master: 0b8f4bce11874dd39985773f5e4a5e49 (127.20.120.194:43333)
I20260621 06:52:31.462877 27100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:55989
WARNING: no policy specified for oryx/127.20.120.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab.
Entry for principal oryx/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab.
I20260621 06:52:31.489080 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45135
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab
--principal=oryx/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:31.815129 27428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:31.815452 27428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:31.815662 27428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:31.823141 27428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:31.823288 27428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:31.823477 27428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:52:31.833288 27428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab
--principal=oryx/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:45135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:31.834713 27428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:31.836072 27428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:31.849617 27433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:31.849577 27434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:31.849665 27436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:31.850469 27428 server_base.cc:1034] running on GCE node
I20260621 06:52:31.851084 27428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:31.852128 27428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:31.853375 27428 hybrid_clock.cc:648] HybridClock initialized: now 1782024751853310 us; error 47 us; skew 500 ppm
I20260621 06:52:31.853722 27428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:31 dist-test-slave-gmf2 krb5kdc[27055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024751, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:31.860070 27428 init.cc:377] Logged in from keytab as oryx/127.20.120.195@KRBTEST.COM (short username oryx)
I20260621 06:52:31.861590 27428 webserver.cc:460] Webserver started at http://127.20.120.195:44367/ using document root <none> and password file <none>
I20260621 06:52:31.862363 27428 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:31.862478 27428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:31.862747 27428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:31.865846 27428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "f7c8fde0dfaf48099d75ed7dec484618"
format_stamp: "Formatted at 2026-06-21 06:52:31 on dist-test-slave-gmf2"
server_key: "7ee1795085f55da5cb72bfe0ce8a0b78"
server_key_iv: "a95aa4d20f9b93a5dc7c1eaaadffc7c1"
server_key_version: "encryptionkey@0"
I20260621 06:52:31.866678 27428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "f7c8fde0dfaf48099d75ed7dec484618"
format_stamp: "Formatted at 2026-06-21 06:52:31 on dist-test-slave-gmf2"
server_key: "7ee1795085f55da5cb72bfe0ce8a0b78"
server_key_iv: "a95aa4d20f9b93a5dc7c1eaaadffc7c1"
server_key_version: "encryptionkey@0"
I20260621 06:52:31.871892 27428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:52:31.875492 27443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:31.876848 27428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 06:52:31.877020 27428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "f7c8fde0dfaf48099d75ed7dec484618"
format_stamp: "Formatted at 2026-06-21 06:52:31 on dist-test-slave-gmf2"
server_key: "7ee1795085f55da5cb72bfe0ce8a0b78"
server_key_iv: "a95aa4d20f9b93a5dc7c1eaaadffc7c1"
server_key_version: "encryptionkey@0"
I20260621 06:52:31.877218 27428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:31.899864 27428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:31.903632 27428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:31.903947 27428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:31.904887 27428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:31.906296 27428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:31.906373 27428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:31.906529 27428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:31.906595 27428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:31.940104 27428 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:43025
I20260621 06:52:31.940135 27555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:43025 every 8 connection(s)
I20260621 06:52:31.941433 27428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:52:31.948556 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 27428
I20260621 06:52:31.948733 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:52:31.949046 27289 heartbeater.cc:503] Master 127.20.120.254:45135 was elected leader, sending a full tablet report...
I20260621 06:52:31.949215 20963 external_mini_cluster.cc:1428] Setting key 54cb537aafdf778fe15895cae4a02152
Jun 21 06:52:31 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024751, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:31.961751 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:31.944072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:59021 (local address 127.20.120.254:45135)
0621 06:52:31.944269 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:31.944276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:31.945378 (+ 1102us) server_negotiation.cc:407] Connection header received
0621 06:52:31.947232 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:31.947236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:31.947309 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:31.947399 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:31.949587 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:31.950871 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:31.952802 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:31.953255 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:31.953353 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:31.956713 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:31.956718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:31.956721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:31.956748 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:31.959270 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:31.959930 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:31.959934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:31.959936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:31.960046 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:31.960389 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:31.960392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:31.960394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:31.960633 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:31.960735 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:31.961339 (+ 604us) server_negotiation.cc:299] Negotiation successful
0621 06:52:31.961492 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 06:52:31.962877 27558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:31.944500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.195:59021)
0621 06:52:31.945214 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:31.945256 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:31.946984 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:31.947604 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:31.947612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:31.948095 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:31.948733 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:31.948760 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:31.951031 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:31.951036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:31.952660 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:31.952671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:31.953442 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:31.953446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:31.953630 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:31.954345 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:31.954367 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:31.956529 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:31.959418 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:31.959424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:31.959435 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:31.959833 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:31.960132 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:31.960135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:31.960138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:31.960304 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:31.960726 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:31.960731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:31.961039 (+ 308us) client_negotiation.cc:769] Sending connection context
0621 06:52:31.961264 (+ 225us) client_negotiation.cc:240] Negotiation successful
0621 06:52:31.961673 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":277,"threads_started":1}
I20260621 06:52:31.964286 27556 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45135
I20260621 06:52:31.964617 27556 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:31.965451 27556 heartbeater.cc:511] Master 127.20.120.254:45135 requested a full tablet report, sending...
I20260621 06:52:31.966789 27100 ts_manager.cc:194] Registered new tserver with Master: f7c8fde0dfaf48099d75ed7dec484618 (127.20.120.195:43025)
I20260621 06:52:31.967502 27100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:59021
I20260621 06:52:31.977025 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.120.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.253.keytab.
Entry for principal oryx/127.20.120.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.253.keytab.
I20260621 06:52:32.020548 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.253:34695
--webserver_interface=127.20.120.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.253.keytab
--principal=oryx/127.20.120.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:45135,127.20.120.253:34695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:32.345001 27561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:32.345284 27561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:32.345436 27561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:32.352463 27561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:32.352576 27561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:32.352666 27561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:32.352761 27561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:32.352831 27561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:32.362452 27561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:45135,127.20.120.253:34695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.253.keytab
--principal=oryx/127.20.120.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.253:34695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.120.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
W20260621 06:52:32.362795 27561 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.120.254:45135,127.20.120.253:34695'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260621 06:52:32.363862 27561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:32.365315 27561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:32.376456 27567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:32.376571 27566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:32.376595 27569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:32.377595 27561 server_base.cc:1034] running on GCE node
I20260621 06:52:32.378393 27561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:32.379716 27561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:32.380955 27561 hybrid_clock.cc:648] HybridClock initialized: now 1782024752380889 us; error 40 us; skew 500 ppm
I20260621 06:52:32.381268 27561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:32 dist-test-slave-gmf2 krb5kdc[27055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024752, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:32.384232 27561 init.cc:377] Logged in from keytab as oryx/127.20.120.253@KRBTEST.COM (short username oryx)
I20260621 06:52:32.385612 27561 webserver.cc:460] Webserver started at http://127.20.120.253:40671/ using document root <none> and password file <none>
I20260621 06:52:32.386317 27561 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:32.386439 27561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:32.386724 27561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:32.389523 27561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data/instance:
uuid: "53b9942a875049a9a62d340b0257ee7b"
format_stamp: "Formatted at 2026-06-21 06:52:32 on dist-test-slave-gmf2"
server_key: "57358d6c7720bbdd3e82790378e5c998"
server_key_iv: "d9579a11beb94fbdccfd2c5b4a425b9d"
server_key_version: "encryptionkey@0"
I20260621 06:52:32.390324 27561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/wal/instance:
uuid: "53b9942a875049a9a62d340b0257ee7b"
format_stamp: "Formatted at 2026-06-21 06:52:32 on dist-test-slave-gmf2"
server_key: "57358d6c7720bbdd3e82790378e5c998"
server_key_iv: "d9579a11beb94fbdccfd2c5b4a425b9d"
server_key_version: "encryptionkey@0"
I20260621 06:52:32.395030 27561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 06:52:32.398095 27576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:32.399330 27561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 06:52:32.399501 27561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/wal
uuid: "53b9942a875049a9a62d340b0257ee7b"
format_stamp: "Formatted at 2026-06-21 06:52:32 on dist-test-slave-gmf2"
server_key: "57358d6c7720bbdd3e82790378e5c998"
server_key_iv: "d9579a11beb94fbdccfd2c5b4a425b9d"
server_key_version: "encryptionkey@0"
I20260621 06:52:32.399705 27561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:32.419988 27561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:32.423887 27561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:32.424129 27561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:32.442080 27561 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.253:34695
I20260621 06:52:32.442086 27627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.253:34695 every 8 connection(s)
I20260621 06:52:32.443444 27561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
I20260621 06:52:32.447304 27628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:32.449339 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 27561
I20260621 06:52:32.454756 27628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:52:32 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.120.253@KRBTEST.COM
I20260621 06:52:32.465421 27423 heartbeater.cc:503] Master 127.20.120.254:45135 was elected leader, sending a full tablet report...
Jun 21 06:52:32 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024752, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.253@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:32.472361 27632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.457065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:46954)
0621 06:52:32.457862 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:32.457932 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:32.458088 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:32.458449 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:32.458454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:32.458994 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:32.459234 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:32.459242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.460533 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.460539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:32.462163 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:32.462180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.462948 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.462957 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:32.463176 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.463823 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:32.463866 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:32.466416 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:32.469756 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:32.469766 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:32.469773 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:32.470269 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:32.470616 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:32.470622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:32.470625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:32.470789 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:32.471444 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:32.471449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:32.471579 (+ 130us) client_negotiation.cc:769] Sending connection context
0621 06:52:32.471734 (+ 155us) client_negotiation.cc:240] Negotiation successful
0621 06:52:32.471968 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":412,"threads_started":1}
I20260621 06:52:32.472401 27631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.451592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.20.120.253:34695)
0621 06:52:32.452489 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:32.452501 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:32.452539 (+ 38us) server_negotiation.cc:407] Connection header received
0621 06:52:32.454135 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:32.454157 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:32.454517 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:32.454871 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:32.456325 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.457709 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.459226 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.460133 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.460311 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.464770 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:32.464780 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:32.464801 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:32.464848 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:32.467997 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.468798 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.468804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.468810 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.468946 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.469326 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.469329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.469331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.469779 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:32.470172 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:32.470549 (+ 377us) server_negotiation.cc:299] Negotiation successful
0621 06:52:32.470943 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":430,"threads_started":1}
I20260621 06:52:32.472422 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.456992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46954 (local address 127.20.120.254:45135)
0621 06:52:32.457181 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:32.457186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:32.458032 (+ 846us) server_negotiation.cc:407] Connection header received
0621 06:52:32.458209 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:32.458215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:32.458301 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:32.458405 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:32.459365 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.460410 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.462329 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.462726 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.462830 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.466588 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:32.466592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:32.466596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:32.466650 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:32.469639 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.470390 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.470395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.470399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.470517 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.470910 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.470915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.470918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.471310 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:32.471481 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:32.471862 (+ 381us) server_negotiation.cc:299] Negotiation successful
0621 06:52:32.472102 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:52:32.474254 27628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:52:32 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024752, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.253@KRBTEST.COM for oryx/127.20.120.253@KRBTEST.COM
I20260621 06:52:32.485399 27632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.475093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34695 (local address 127.0.0.1:36196)
0621 06:52:32.475237 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:32.475252 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:32.475367 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:32.475617 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:32.475621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:32.475923 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:32.476033 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:32.476052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.476907 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.476911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:32.477714 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:32.477723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.478421 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.478426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:32.478667 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.479094 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:32.479109 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:32.480810 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:32.483387 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:32.483394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:32.483397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:32.483704 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:32.483993 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:32.483999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:32.484002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:32.484113 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:32.484553 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:32.484557 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:32.484669 (+ 112us) client_negotiation.cc:769] Sending connection context
0621 06:52:32.484783 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 06:52:32.485035 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":35}
I20260621 06:52:32.485409 27631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.475021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.20.120.253:34695)
0621 06:52:32.475182 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:32.475188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:32.475319 (+ 131us) server_negotiation.cc:407] Connection header received
0621 06:52:32.475466 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:32.475471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:32.475523 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:32.475605 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:32.476153 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.476756 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.477840 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.478293 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.478415 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.480983 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:32.480987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:32.480990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:32.481014 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:32.483277 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.483809 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.483813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.483816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.483905 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.484227 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.484233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.484236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.484457 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:32.484581 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:32.484853 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 06:52:32.485142 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":41}
I20260621 06:52:32.489854 27628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: Bootstrap starting.
I20260621 06:52:32.492851 27628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:32.493824 27628 log.cc:824] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:32.496105 27628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: No bootstrap required, opened a new log
I20260621 06:52:32.502362 27628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } }
I20260621 06:52:32.502661 27628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:32.502750 27628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b9942a875049a9a62d340b0257ee7b, State: Initialized, Role: FOLLOWER
I20260621 06:52:32.503269 27628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } }
I20260621 06:52:32.504019 27634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } } }
I20260621 06:52:32.504289 27634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:52:32.504854 27628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: configured and running, proceeding with master startup.
W20260621 06:52:32.512686 27645 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 06:52:32.512820 27645 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 06:52:32.514171 27561 master_runner.cc:428] Detected that this master 53b9942a875049a9a62d340b0257ee7b is joining an existing cluster
I20260621 06:52:32.514257 27561 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.120.253:34695
I20260621 06:52:32.531823 27634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:52:32.532132 27634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } }
I20260621 06:52:32.533167 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.517953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.20.120.254:45135)
0621 06:52:32.518129 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:32.518137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:32.519752 (+ 1615us) server_negotiation.cc:407] Connection header received
0621 06:52:32.519913 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:32.519919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:32.519997 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:32.520098 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:32.521250 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.522184 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.523564 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.524145 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.524276 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.526647 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:32.526653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:32.526658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:32.526695 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:32.530046 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.530861 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.530867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.530871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.531015 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.531485 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.531491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.531494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.531835 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:32.531958 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:32.532568 (+ 610us) server_negotiation.cc:299] Negotiation successful
0621 06:52:32.532785 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 06:52:32.533471 27634 leader_election.cc:288] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85e295b70bac43c0a54c15c239835d0d (127.20.120.254:45135)
I20260621 06:52:32.534150 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.518336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:46958)
0621 06:52:32.519656 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:32.519671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:32.519776 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:32.520137 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:32.520147 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:32.520816 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:32.521113 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:32.521124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.522329 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.522333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:32.523427 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:32.523439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.524340 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.524348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:32.524522 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.525213 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:32.525235 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:32.526454 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:32.530211 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:32.530216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:32.530219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:32.530716 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:32.531165 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:32.531169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:32.531173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:32.531355 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:32.532073 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:32.532078 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:32.532269 (+ 191us) client_negotiation.cc:769] Sending connection context
0621 06:52:32.533379 (+ 1110us) client_negotiation.cc:240] Negotiation successful
0621 06:52:32.533659 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"mutex_wait_us":2,"thread_start_us":1038,"threads_started":1}
I20260621 06:52:32.534350 27110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53b9942a875049a9a62d340b0257ee7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e295b70bac43c0a54c15c239835d0d" is_pre_election: true
I20260621 06:52:32.535404 27580 leader_election.cc:302] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 53b9942a875049a9a62d340b0257ee7b; no voters: 85e295b70bac43c0a54c15c239835d0d
I20260621 06:52:32.535818 27634 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 06:52:32.536460 27631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.518079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.20.120.253:34695)
0621 06:52:32.518273 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:32.518279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:32.519086 (+ 807us) server_negotiation.cc:407] Connection header received
0621 06:52:32.519256 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:32.519286 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:32.519402 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:32.519515 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:32.520817 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.521750 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.523745 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.524437 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.524597 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.526706 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:32.526711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:32.526717 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:32.526748 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:32.530580 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.531372 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.531390 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.531395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.531514 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.534375 (+ 2861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.534392 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.534395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.534774 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:32.534936 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:32.535378 (+ 442us) server_negotiation.cc:299] Negotiation successful
0621 06:52:32.535653 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":48}
I20260621 06:52:32.537551 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.518312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34695 (local address 127.0.0.1:36212)
0621 06:52:32.518977 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:32.518995 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:32.519132 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:32.519692 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:32.519695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:32.520358 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:32.520679 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:32.520687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.521970 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.521975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:32.523612 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:32.523630 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.524607 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.524612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:32.524794 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.525408 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:32.525432 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:32.526533 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:32.530709 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:32.530716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:32.530723 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:32.531253 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:32.533941 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:32.533945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:32.533948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:32.534123 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:32.535000 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:32.535006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:32.535137 (+ 131us) client_negotiation.cc:769] Sending connection context
0621 06:52:32.536790 (+ 1653us) client_negotiation.cc:240] Negotiation successful
0621 06:52:32.537110 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":3,"thread_start_us":394,"threads_started":1}
WARNING: no policy specified for oryx/127.20.120.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.252@KRBTEST.COM" created.
Jun 21 06:52:32 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024750, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.254@KRBTEST.COM for oryx/127.20.120.253@KRBTEST.COM
I20260621 06:52:32.562683 27654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.543215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34695 (local address 127.0.0.1:36222)
0621 06:52:32.543923 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:32.543966 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:32.544134 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:32.544622 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:32.544629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:32.545410 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:32.545669 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:32.545681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.547264 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.547270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:32.548846 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:32.548860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.549998 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.550006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:32.550227 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.550896 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:32.550919 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:32.555436 (+ 4517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:32.559839 (+ 4403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:32.559850 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:32.559858 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:32.560376 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:32.560717 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:32.560722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:32.560725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:32.560862 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:32.561350 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:32.561358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:32.561569 (+ 211us) client_negotiation.cc:769] Sending connection context
0621 06:52:32.561754 (+ 185us) client_negotiation.cc:240] Negotiation successful
0621 06:52:32.562127 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":293,"threads_started":1}
I20260621 06:52:32.562703 27631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.543243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.20.120.253:34695)
0621 06:52:32.543425 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:32.543433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:32.544054 (+ 621us) server_negotiation.cc:407] Connection header received
0621 06:52:32.544312 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:32.544320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:32.544454 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:32.544561 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:32.545847 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.547126 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.549000 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.549792 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.549968 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.555633 (+ 5665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:32.555642 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:32.555650 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:32.555691 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:32.559697 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.560477 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.560481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.560483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.560596 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.560972 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.560977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.560980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.561223 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:32.561376 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:32.562098 (+ 722us) server_negotiation.cc:299] Negotiation successful
0621 06:52:32.562314 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 06:52:32.564766 27100 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.120.253:34695: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "53b9942a875049a9a62d340b0257ee7b"
member_type: NON_VOTER
last_known_addr {
host: "127.20.120.253"
port: 34695
}
attrs {
promote: true
}
}
dest_uuid: "85e295b70bac43c0a54c15c239835d0d"
cas_config_opid_index: -1
I20260621 06:52:32.565979 27100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true } }
I20260621 06:52:32.578529 27657 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 53b9942a875049a9a62d340b0257ee7b (127.20.120.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true } } }
I20260621 06:52:32.579981 27603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:32.582309 27657 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.120.253:34695
I20260621 06:52:32.582738 27657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true } } }
I20260621 06:52:32.582971 27657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:32.583006 27561 master.cc:537] Master@127.20.120.253:34695 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.252.keytab.
Entry for principal oryx/127.20.120.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.252.keytab.
I20260621 06:52:32.584496 27603 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 FOLLOWER]: Refusing update from remote peer 85e295b70bac43c0a54c15c239835d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260621 06:52:32.585386 27657 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:32.585986 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.252:34201
--webserver_interface=127.20.120.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.252.keytab
--principal=oryx/127.20.120.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:45135,127.20.120.253:34695,127.20.120.252:34201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:32.588651 27085 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260621 06:52:32.594803 27561 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:32.595288 27561 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:32.595498 27561 tablet_replica.cc:330] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: stopping tablet replica
W20260621 06:52:32.600517 27085 consensus_peers.cc:487] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d -> Peer 53b9942a875049a9a62d340b0257ee7b (127.20.120.253:34695): Couldn't send request to peer 53b9942a875049a9a62d340b0257ee7b. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260621 06:52:32.602749 27561 master.cc:552] Master@127.20.120.253:34695 shutdown complete.
I20260621 06:52:32.602856 27561 master_runner.cc:305] Clearing existing system tablet
I20260621 06:52:32.603238 27561 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:32.603338 27561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:32.605537 27561 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260621 06:52:32.607134 27663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:32.607426 27561 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:32.607764 27561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 06:52:32.607875 27561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/wal
uuid: "53b9942a875049a9a62d340b0257ee7b"
format_stamp: "Formatted at 2026-06-21 06:52:32 on dist-test-slave-gmf2"
server_key: "57358d6c7720bbdd3e82790378e5c998"
server_key_iv: "d9579a11beb94fbdccfd2c5b4a425b9d"
server_key_version: "encryptionkey@0"
I20260621 06:52:32.611169 27561 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:32.615895 27561 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 06:52:32.616070 27561 log.cc:1196] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260621 06:52:32.616443 27561 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: Deleting consensus metadata
I20260621 06:52:32.616681 27561 master_runner.cc:315] Copying system tablet from 127.20.120.254:45135
I20260621 06:52:32.621461 27561 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.120.254:45135
I20260621 06:52:32.634394 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.622320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.20.120.254:45135)
0621 06:52:32.622555 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:32.622565 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:32.623030 (+ 465us) server_negotiation.cc:407] Connection header received
0621 06:52:32.623226 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:32.623232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:32.623318 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:32.623423 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:32.624237 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.624936 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.626215 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.626643 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.626775 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.628520 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:32.628526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:32.628536 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:32.628566 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:32.631788 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.632454 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.632462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.632467 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.632588 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:32.632964 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:32.632971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:32.632976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:32.633247 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:32.633413 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:32.633760 (+ 347us) server_negotiation.cc:299] Negotiation successful
0621 06:52:32.634029 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 06:52:32.634816 27669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:32.622277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:46974)
0621 06:52:32.622912 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:32.622930 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:32.623097 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:32.623448 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:32.623452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:32.623870 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:32.624079 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:32.624090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.625077 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.625090 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:32.626076 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:32.626088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:32.626798 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:32.626805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:32.626994 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:32.627617 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:32.627634 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:32.628323 (+ 689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:32.631961 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:32.631968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:32.631972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:32.632333 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:32.632699 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:32.632704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:32.632708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:32.632860 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:32.633370 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:32.633377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:32.633531 (+ 154us) client_negotiation.cc:769] Sending connection context
0621 06:52:32.633675 (+ 144us) client_negotiation.cc:240] Negotiation successful
0621 06:52:32.634001 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":431,"threads_started":1}
I20260621 06:52:32.635843 27120 tablet_copy_service.cc:140] P 85e295b70bac43c0a54c15c239835d0d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 53b9942a875049a9a62d340b0257ee7b ({username='oryx', principal='oryx/127.20.120.253@KRBTEST.COM'} at 127.0.0.1:46974)
I20260621 06:52:32.636094 27120 tablet_copy_service.cc:161] P 85e295b70bac43c0a54c15c239835d0d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 53b9942a875049a9a62d340b0257ee7b at {username='oryx', principal='oryx/127.20.120.253@KRBTEST.COM'} at 127.0.0.1:46974: session id = 53b9942a875049a9a62d340b0257ee7b-00000000000000000000000000000000
I20260621 06:52:32.639205 27120 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 06:52:32.640978 27561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 06:52:32.646242 27561 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: tablet copy: Starting download of 0 data blocks...
I20260621 06:52:32.646521 27561 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: tablet copy: Starting download of 1 WAL segments...
I20260621 06:52:32.648705 27561 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 06:52:32.651839 27120 tablet_copy_service.cc:342] P 85e295b70bac43c0a54c15c239835d0d: Request end of tablet copy session 53b9942a875049a9a62d340b0257ee7b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.120.253@KRBTEST.COM'} at 127.0.0.1:46974
I20260621 06:52:32.652019 27120 tablet_copy_service.cc:433] P 85e295b70bac43c0a54c15c239835d0d: ending tablet copy session 53b9942a875049a9a62d340b0257ee7b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 53b9942a875049a9a62d340b0257ee7b
I20260621 06:52:32.661479 27561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:32.664013 27670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:32.669701 27671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:32.674258 27561 server_base.cc:1034] running on GCE node
W20260621 06:52:32.674846 27673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:32.675334 27561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:32.675948 27561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:32.677150 27561 hybrid_clock.cc:648] HybridClock initialized: now 1782024752677079 us; error 48 us; skew 500 ppm
I20260621 06:52:32.677369 27561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:32 dist-test-slave-gmf2 krb5kdc[27055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024752, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:32.679227 27561 init.cc:377] Logged in from keytab as oryx/127.20.120.253@KRBTEST.COM (short username oryx)
I20260621 06:52:32.680374 27561 webserver.cc:460] Webserver started at http://127.20.120.253:34167/ using document root <none> and password file <none>
I20260621 06:52:32.680702 27561 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:32.680799 27561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:32.683187 27561 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 06:52:32.684737 27680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:32.685119 27561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 06:52:32.685351 27561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/wal
uuid: "53b9942a875049a9a62d340b0257ee7b"
format_stamp: "Formatted at 2026-06-21 06:52:32 on dist-test-slave-gmf2"
server_key: "57358d6c7720bbdd3e82790378e5c998"
server_key_iv: "d9579a11beb94fbdccfd2c5b4a425b9d"
server_key_version: "encryptionkey@0"
I20260621 06:52:32.685508 27561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:32.707078 27561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:32.710320 27561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:32.734748 27561 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.253:34695
I20260621 06:52:32.734782 27731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.253:34695 every 8 connection(s)
I20260621 06:52:32.735719 27561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
I20260621 06:52:32.742444 27732 sys_catalog.cc:263] Verifying existing consensus state
I20260621 06:52:32.747490 27732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: Bootstrap starting.
I20260621 06:52:32.760900 27732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 06:52:32.761368 27732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: Bootstrap complete.
I20260621 06:52:32.762210 27732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true } }
I20260621 06:52:32.762416 27732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 53b9942a875049a9a62d340b0257ee7b, State: Initialized, Role: LEARNER
I20260621 06:52:32.762633 27732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true } }
I20260621 06:52:32.763473 27735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true } } }
I20260621 06:52:32.763787 27732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:32.763952 27735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: This master's current role is: LEARNER
I20260621 06:52:32.771641 27746 catalog_manager.cc:1259] Loaded cluster ID: 2660fb547c4648959bd33249505d04a5
I20260621 06:52:32.771744 27746 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: loading cluster ID for follower catalog manager: success
I20260621 06:52:32.774086 27746 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: acquiring CA information for follower catalog manager: success
I20260621 06:52:32.775857 27746 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 06:52:32.969759 27556 heartbeater.cc:503] Master 127.20.120.254:45135 was elected leader, sending a full tablet report...
W20260621 06:52:33.052853 27661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:33.053225 27661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:33.053367 27661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:33.065588 27661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:33.065711 27661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:33.065795 27661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:33.065881 27661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:33.065963 27661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:33.082239 27661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:45135,127.20.120.253:34695,127.20.120.252:34201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.252.keytab
--principal=oryx/127.20.120.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.252:34201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.120.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:33.083868 27661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:33.085969 27661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 06:52:33.096714 27751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.073068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.20.120.253:34695)
0621 06:52:33.089232 (+ 16164us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.089238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.089260 (+ 22us) server_negotiation.cc:407] Connection header received
0621 06:52:33.089459 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.089463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.089529 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.089660 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:33.090172 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.091273 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.093526 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.095352 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.095518 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.095680 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.096216 (+ 536us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.096356 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16015,"thread_start_us":16194,"threads_started":1}
I20260621 06:52:33.097402 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.072914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34695 (local address 127.0.0.1:36234)
0621 06:52:33.088773 (+ 15859us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.088814 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.089268 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.089709 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.089716 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:33.089748 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:33.090033 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.090044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.091416 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.091421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.093359 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.093373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.095835 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.095840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.096016 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.096070 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 06:52:33.096983 (+ 913us) client_negotiation.cc:240] Negotiation successful
0621 06:52:33.097131 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15666,"thread_start_us":15979,"threads_started":1}
I20260621 06:52:33.098678 27707 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260621 06:52:33.099545 27735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e295b70bac43c0a54c15c239835d0d. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true } } }
I20260621 06:52:33.099772 27735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: This master's current role is: LEARNER
W20260621 06:52:33.102236 27754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:33.102237 27753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:33.103742 27661 server_base.cc:1034] running on GCE node
W20260621 06:52:33.103971 27756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:33.104775 27661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:33.106240 27661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:33.107462 27661 hybrid_clock.cc:648] HybridClock initialized: now 1782024753107420 us; error 37 us; skew 500 ppm
I20260621 06:52:33.107788 27661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:33 dist-test-slave-gmf2 krb5kdc[27055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024753, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:33.111451 27661 init.cc:377] Logged in from keytab as oryx/127.20.120.252@KRBTEST.COM (short username oryx)
I20260621 06:52:33.112958 27661 webserver.cc:460] Webserver started at http://127.20.120.252:35747/ using document root <none> and password file <none>
I20260621 06:52:33.113631 27661 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:33.113768 27661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:33.114101 27661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:33.116981 27661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data/instance:
uuid: "05e64346b296436689045e4af52dd145"
format_stamp: "Formatted at 2026-06-21 06:52:33 on dist-test-slave-gmf2"
server_key: "daa0aef2a77b8f6286a1ae84952c9705"
server_key_iv: "93d59288e97754fdfa4b424815a2a406"
server_key_version: "encryptionkey@0"
I20260621 06:52:33.117854 27661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/wal/instance:
uuid: "05e64346b296436689045e4af52dd145"
format_stamp: "Formatted at 2026-06-21 06:52:33 on dist-test-slave-gmf2"
server_key: "daa0aef2a77b8f6286a1ae84952c9705"
server_key_iv: "93d59288e97754fdfa4b424815a2a406"
server_key_version: "encryptionkey@0"
I20260621 06:52:33.122860 27661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 06:52:33.126019 27763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:33.127255 27661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:52:33.127435 27661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/wal
uuid: "05e64346b296436689045e4af52dd145"
format_stamp: "Formatted at 2026-06-21 06:52:33 on dist-test-slave-gmf2"
server_key: "daa0aef2a77b8f6286a1ae84952c9705"
server_key_iv: "93d59288e97754fdfa4b424815a2a406"
server_key_version: "encryptionkey@0"
I20260621 06:52:33.127621 27661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:33.148355 27661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:33.152284 27661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:33.152557 27661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:33.171422 27661 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.252:34201
I20260621 06:52:33.171445 27814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.252:34201 every 8 connection(s)
I20260621 06:52:33.172758 27661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
I20260621 06:52:33.176363 27815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:33.180706 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 27661
I20260621 06:52:33.182977 27815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:52:33 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024753, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.252@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
Jun 21 06:52:33 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.120.252@KRBTEST.COM
I20260621 06:52:33.202842 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.185639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.20.120.254:45135)
0621 06:52:33.186197 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.186201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.186987 (+ 786us) server_negotiation.cc:407] Connection header received
0621 06:52:33.187054 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.187059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.187128 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.187263 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.188820 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.189990 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.192156 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.192747 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.192863 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.196164 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:33.196170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:33.196173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:33.196204 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:33.199450 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.200202 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.200207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.200211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.200361 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.200783 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.200790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.200796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.201192 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:33.201351 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.202221 (+ 870us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.202412 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":318,"threads_started":1}
I20260621 06:52:33.203764 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.186032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:46984)
0621 06:52:33.186726 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.186770 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.186945 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.187336 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.187343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:33.188161 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:33.188657 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.188676 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.190142 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.190150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.192022 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.192040 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.192967 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.192973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.193139 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.193852 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:33.193879 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:33.195992 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:33.199565 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.199574 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.199583 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.200091 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.200470 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.200473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.200476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.200648 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.201315 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:33.201322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:33.201743 (+ 421us) client_negotiation.cc:769] Sending connection context
0621 06:52:33.202105 (+ 362us) client_negotiation.cc:240] Negotiation successful
0621 06:52:33.202588 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":301,"threads_started":1}
I20260621 06:52:33.204527 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.182726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44418 (local address 127.20.120.252:34201)
0621 06:52:33.183487 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.183498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.183539 (+ 41us) server_negotiation.cc:407] Connection header received
0621 06:52:33.185266 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.185294 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.186227 (+ 933us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.186677 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.188496 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.190237 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.191982 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.192721 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.192889 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.197614 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:33.197626 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:33.197644 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:33.197691 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:33.201098 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.201972 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.201978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.201980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.202141 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.202574 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.202580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.202584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.203007 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:33.203302 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.203843 (+ 541us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.204061 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"spinlock_wait_cycles":14080,"thread_start_us":368,"threads_started":1}
I20260621 06:52:33.205534 27815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:52:33 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024753, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.252@KRBTEST.COM for oryx/127.20.120.253@KRBTEST.COM
I20260621 06:52:33.217943 27751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.206339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.20.120.253:34695)
0621 06:52:33.206534 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.206542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.206668 (+ 126us) server_negotiation.cc:407] Connection header received
0621 06:52:33.206867 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.206871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.206931 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.207028 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.207671 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.208346 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.209516 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.209966 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.210086 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.212802 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:33.212807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:33.212811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:33.212840 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:33.215607 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.216191 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.216197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.216200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.216318 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.216650 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.216656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.216659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.216908 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:33.217051 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.217381 (+ 330us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.217587 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 06:52:33.217885 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.206401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34695 (local address 127.0.0.1:36244)
0621 06:52:33.206585 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.206601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.206717 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.207043 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.207047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:33.207409 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:33.207529 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.207536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.208462 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.208470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.209386 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.209396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.210094 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.210099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.210271 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.210824 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:33.210839 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:33.212587 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:33.215729 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.215736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.215740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.216079 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.216420 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.216426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.216430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.216551 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.217018 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:33.217023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:33.217155 (+ 132us) client_negotiation.cc:769] Sending connection context
0621 06:52:33.217285 (+ 130us) client_negotiation.cc:240] Negotiation successful
0621 06:52:33.217553 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 06:52:33.218905 27815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:52:33 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024753, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.252@KRBTEST.COM for oryx/127.20.120.252@KRBTEST.COM
I20260621 06:52:33.229853 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.219492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:34201 (local address 127.0.0.1:44424)
0621 06:52:33.219645 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.219659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.219745 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.219972 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.219975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:33.220263 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:33.220394 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.220401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.221264 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.221268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.222137 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.222148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.222768 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.222777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.222990 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.223535 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:33.223551 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:33.225347 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:33.227937 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.227946 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.227949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.228287 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.228571 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.228574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.228577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.228718 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.229154 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:33.229157 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:33.229279 (+ 122us) client_negotiation.cc:769] Sending connection context
0621 06:52:33.229393 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 06:52:33.229571 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260621 06:52:33.229965 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.219639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.20.120.252:34201)
0621 06:52:33.219770 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.219774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.219787 (+ 13us) server_negotiation.cc:407] Connection header received
0621 06:52:33.219826 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.219829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.219874 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.219944 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.220510 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.221161 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.222244 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.222619 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.222727 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.225502 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:33.225506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:33.225509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:33.225535 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:33.227826 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.228375 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.228379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.228381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.228489 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.228807 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.228810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.228812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.229040 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:33.229141 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.229463 (+ 322us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.229688 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":45}
I20260621 06:52:33.234850 27815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: Bootstrap starting.
I20260621 06:52:33.238090 27815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:33.239485 27815 log.cc:824] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:33.242072 27815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: No bootstrap required, opened a new log
I20260621 06:52:33.249202 27815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } }
I20260621 06:52:33.249516 27815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:33.249624 27815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e64346b296436689045e4af52dd145, State: Initialized, Role: FOLLOWER
I20260621 06:52:33.250258 27815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } }
I20260621 06:52:33.251161 27822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } } }
I20260621 06:52:33.251472 27822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:52:33.252280 27815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:33.258991 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 27161
W20260621 06:52:33.262719 27833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 06:52:33.262844 27833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 06:52:33.266541 27661 master_runner.cc:428] Detected that this master 05e64346b296436689045e4af52dd145 is joining an existing cluster
I20260621 06:52:33.266683 27661 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.120.252:34201
Jun 21 06:52:33 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.120.193@KRBTEST.COM
I20260621 06:52:33.284430 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.261077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.20.120.193:40259)
0621 06:52:33.262064 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.262073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.262100 (+ 27us) server_negotiation.cc:407] Connection header received
0621 06:52:33.262166 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.262174 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.262440 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.264301 (+ 1861us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.264562 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.266070 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.267946 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.269078 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.269248 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.275302 (+ 6054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:33.275317 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:33.275335 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:33.275380 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:33.279398 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.280356 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.280368 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.280374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.280529 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.281072 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.281078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.281083 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.282167 (+ 1084us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:33.282450 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.283642 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.283951 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":637,"threads_started":1}
I20260621 06:52:33.286487 27284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:33.294445 27751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.273207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.20.120.253:34695)
0621 06:52:33.273408 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.273416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.274066 (+ 650us) server_negotiation.cc:407] Connection header received
0621 06:52:33.274324 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.274331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.274424 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.274502 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.276134 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.277299 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.279480 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.280110 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.280211 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.283122 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:33.283129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:33.283136 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:33.283174 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:33.286646 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.291115 (+ 4469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.291125 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.291131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.291372 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.292065 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.292073 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.292079 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.292511 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:33.292671 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.293559 (+ 888us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.293923 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 06:52:33.296247 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.273314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34695 (local address 127.0.0.1:36254)
0621 06:52:33.273962 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.273984 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.274176 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.274729 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.274736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:33.275477 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:33.275859 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.275870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.277506 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.277517 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.279291 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.279308 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.280541 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.280546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.280754 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.281746 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:33.281767 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:33.282934 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:33.290063 (+ 7129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.290071 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.290077 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.290800 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.291574 (+ 774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.291582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.291588 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.291890 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.292787 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:33.292799 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:33.293103 (+ 304us) client_negotiation.cc:769] Sending connection context
0621 06:52:33.295092 (+ 1989us) client_negotiation.cc:240] Negotiation successful
0621 06:52:33.295552 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":320,"threads_started":1}
I20260621 06:52:33.311730 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.274402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.20.120.252:34201)
0621 06:52:33.274542 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.274548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.289860 (+ 15312us) server_negotiation.cc:407] Connection header received
0621 06:52:33.292741 (+ 2881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.292752 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.292829 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.292908 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.297137 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.298207 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.299929 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.300631 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.300710 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.304905 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:33.304911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:33.304915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:33.304945 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:33.308741 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.309491 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.309495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.309498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.309632 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.310023 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.310029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.310032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.310451 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:33.310595 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.311094 (+ 499us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.311340 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 06:52:33.313208 27841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.274125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:34201 (local address 127.0.0.1:44436)
0621 06:52:33.289735 (+ 15610us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.289755 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.289919 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.293111 (+ 3192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.293115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:33.294179 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:33.294501 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.294512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.298411 (+ 3899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.298415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.299776 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.299789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.300950 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.300955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.301108 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.301721 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:33.301762 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:33.304738 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:33.308890 (+ 4152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.308895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.308899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.309360 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.309758 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.309762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.309766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.309898 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.310664 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:33.310669 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:33.310830 (+ 161us) client_negotiation.cc:769] Sending connection context
0621 06:52:33.312314 (+ 1484us) client_negotiation.cc:240] Negotiation successful
0621 06:52:33.312662 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15448,"thread_start_us":16599,"threads_started":1}
I20260621 06:52:33.316618 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.279987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.20.120.254:45135)
0621 06:52:33.280161 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.280167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.290520 (+ 10353us) server_negotiation.cc:407] Connection header received
0621 06:52:33.291068 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.291075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.291158 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.291230 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.294679 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.295742 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.298325 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.299508 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.299654 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.303100 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:33.303109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:33.303118 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:33.303163 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:33.307345 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.313942 (+ 6597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.313950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.313955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.314145 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.314692 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.314698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.314703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.315094 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:33.315286 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.316075 (+ 789us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.316298 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:52:33.317852 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.280558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:46996)
0621 06:52:33.290411 (+ 9853us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.290433 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.290885 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.291444 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.291449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:33.292187 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:33.292547 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.292561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.295929 (+ 3368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.295933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.298137 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.298150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.300017 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.300026 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.300307 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.301261 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:33.301295 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:33.302866 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:33.307544 (+ 4678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.307549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.307554 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.313753 (+ 6199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.314315 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.314321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.314326 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.314540 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.315379 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:33.315395 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:33.315621 (+ 226us) client_negotiation.cc:769] Sending connection context
0621 06:52:33.317083 (+ 1462us) client_negotiation.cc:240] Negotiation successful
0621 06:52:33.317421 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9724,"thread_start_us":9980,"threads_started":1}
Jun 21 06:52:33 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024750, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.254@KRBTEST.COM for oryx/127.20.120.252@KRBTEST.COM
I20260621 06:52:33.340173 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.323466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:34201 (local address 127.0.0.1:44452)
0621 06:52:33.323834 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.323854 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.323958 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.325087 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.325093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:33.325535 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:33.325716 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.325727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.327176 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.327182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.328544 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.328559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.329591 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.329597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.329773 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.330430 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:33.330457 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:33.333216 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:33.337566 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.337572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.337576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.338095 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.338508 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.338514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.338518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.338696 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.339269 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:33.339274 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:33.339435 (+ 161us) client_negotiation.cc:769] Sending connection context
0621 06:52:33.339576 (+ 141us) client_negotiation.cc:240] Negotiation successful
0621 06:52:33.339830 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222}
I20260621 06:52:33.340533 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.324542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.20.120.252:34201)
0621 06:52:33.324718 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.324724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.324741 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:52:33.324835 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.324840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.324939 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.325034 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.325870 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.327039 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.328694 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.329426 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.329541 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.333421 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:33.333428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:33.333433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:33.333478 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:33.337418 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.338222 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.338227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.338231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.338393 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.338813 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.338819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.338824 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.339149 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:33.339289 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.339680 (+ 391us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.339909 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 06:52:33.342139 27100 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.120.252:34201: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "05e64346b296436689045e4af52dd145"
member_type: NON_VOTER
last_known_addr {
host: "127.20.120.252"
port: 34201
}
attrs {
promote: true
}
}
dest_uuid: "85e295b70bac43c0a54c15c239835d0d"
cas_config_opid_index: 5
I20260621 06:52:33.343142 27100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } }
I20260621 06:52:33.344640 27657 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 05e64346b296436689045e4af52dd145 (127.20.120.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } } }
I20260621 06:52:33.345294 27790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:33.348932 27790 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 FOLLOWER]: Refusing update from remote peer 85e295b70bac43c0a54c15c239835d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 06:52:33.361112 27657 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.120.252:34201
I20260621 06:52:33.361865 27661 master.cc:537] Master@127.20.120.252:34201 shutting down...
I20260621 06:52:33.369544 27846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:33.371883 27707 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 LEARNER]: Refusing update from remote peer 85e295b70bac43c0a54c15c239835d0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 06:52:33.374027 27657 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260621 06:52:33.374277 27844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } } }
I20260621 06:52:33.374730 27844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:33.387586 27707 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 05e64346b296436689045e4af52dd145 (127.20.120.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } } }
I20260621 06:52:33.390679 27735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: NON_VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: true } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } } }
I20260621 06:52:33.390983 27735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: This master's current role is: LEARNER
I20260621 06:52:33.391745 27844 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d: attempting to promote NON_VOTER 53b9942a875049a9a62d340b0257ee7b to VOTER
I20260621 06:52:33.392422 27844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } }
I20260621 06:52:33.394407 27707 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 LEARNER]: Refusing update from remote peer 85e295b70bac43c0a54c15c239835d0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260621 06:52:33.394919 27844 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260621 06:52:33.396901 27657 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 53b9942a875049a9a62d340b0257ee7b (127.20.120.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } } }
I20260621 06:52:33.399796 27657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } } }
I20260621 06:52:33.400025 27657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:33.400280 27846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } } }
I20260621 06:52:33.400579 27846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:33.402171 27707 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 53b9942a875049a9a62d340b0257ee7b (127.20.120.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } } }
I20260621 06:52:33.403071 27661 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:33.403527 27661 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:33.403733 27661 tablet_replica.cc:330] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: stopping tablet replica
I20260621 06:52:33.407089 27735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } } }
I20260621 06:52:33.407333 27735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:52:33.419327 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.418898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:34201 (local address 127.0.0.1:44468)
0621 06:52:33.419068 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.419166 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:34201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260621 06:52:33.419690 27085 consensus_peers.cc:487] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d -> Peer 05e64346b296436689045e4af52dd145 (127.20.120.252:34201): Couldn't send request to peer 05e64346b296436689045e4af52dd145. Status: Network error: Client connection negotiation failed: client connection to 127.20.120.252:34201: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 06:52:33.421104 27661 master.cc:552] Master@127.20.120.252:34201 shutdown complete.
I20260621 06:52:33.421204 27661 master_runner.cc:305] Clearing existing system tablet
I20260621 06:52:33.421521 27661 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:33.421641 27661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:33.423882 27661 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260621 06:52:33.425243 27851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:33.425559 27661 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:33.425824 27661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 06:52:33.425951 27661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/wal
uuid: "05e64346b296436689045e4af52dd145"
format_stamp: "Formatted at 2026-06-21 06:52:33 on dist-test-slave-gmf2"
server_key: "daa0aef2a77b8f6286a1ae84952c9705"
server_key_iv: "93d59288e97754fdfa4b424815a2a406"
server_key_version: "encryptionkey@0"
I20260621 06:52:33.429221 27661 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:33.439277 27661 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 06:52:33.439469 27661 log.cc:1196] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260621 06:52:33.439951 27661 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: Deleting consensus metadata
I20260621 06:52:33.440294 27661 master_runner.cc:315] Copying system tablet from 127.20.120.254:45135
I20260621 06:52:33.442667 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 27161
I20260621 06:52:33.454458 27661 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: tablet copy: Beginning tablet copy session from remote peer at address 127.20.120.254:45135
I20260621 06:52:33.466655 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:40259
--local_ip_for_outbound_sockets=127.20.120.193
--tserver_master_addrs=127.20.120.254:45135
--webserver_port=42851
--webserver_interface=127.20.120.193
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab
--principal=oryx/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.120.254:45135,127.20.120.253:34695
--tserver_master_addrs=127.20.120.254:45135,127.20.120.253:34695,127.20.120.252:34201 with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:52:33.474534 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.455309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.20.120.254:45135)
0621 06:52:33.455567 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.455575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.458534 (+ 2959us) server_negotiation.cc:407] Connection header received
0621 06:52:33.458760 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.458767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.458852 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.458964 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.460162 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.461213 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.463519 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.464330 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.464499 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.467726 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:33.467735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:33.467744 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:33.467789 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:33.471487 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.472279 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.472287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.472293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.472447 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.472882 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.472890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.472896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.473322 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:33.473471 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.473938 (+ 467us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.474172 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260621 06:52:33.474543 27857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.455301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:47000)
0621 06:52:33.458399 (+ 3098us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.458421 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.458588 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.459021 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.459029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:33.459678 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:33.459981 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.459997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.461388 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.461396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.463273 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.463294 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.464865 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.464873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.465103 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.466106 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:33.466134 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:33.467483 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:33.471618 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.471624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.471628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.472136 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.472560 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.472564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.472568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.472759 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.473442 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:33.473447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:33.473625 (+ 178us) client_negotiation.cc:769] Sending connection context
0621 06:52:33.473759 (+ 134us) client_negotiation.cc:240] Negotiation successful
0621 06:52:33.474095 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2914,"thread_start_us":2756,"threads_started":1}
I20260621 06:52:33.475162 27120 tablet_copy_service.cc:140] P 85e295b70bac43c0a54c15c239835d0d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 05e64346b296436689045e4af52dd145 ({username='oryx', principal='oryx/127.20.120.252@KRBTEST.COM'} at 127.0.0.1:47000)
I20260621 06:52:33.475324 27120 tablet_copy_service.cc:161] P 85e295b70bac43c0a54c15c239835d0d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 05e64346b296436689045e4af52dd145 at {username='oryx', principal='oryx/127.20.120.252@KRBTEST.COM'} at 127.0.0.1:47000: session id = 05e64346b296436689045e4af52dd145-00000000000000000000000000000000
I20260621 06:52:33.478966 27120 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 06:52:33.481309 27661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 06:52:33.489315 27661 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: tablet copy: Starting download of 0 data blocks...
I20260621 06:52:33.489641 27661 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: tablet copy: Starting download of 1 WAL segments...
I20260621 06:52:33.492148 27661 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 06:52:33.496372 27120 tablet_copy_service.cc:342] P 85e295b70bac43c0a54c15c239835d0d: Request end of tablet copy session 05e64346b296436689045e4af52dd145-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.120.252@KRBTEST.COM'} at 127.0.0.1:47000
I20260621 06:52:33.496500 27120 tablet_copy_service.cc:433] P 85e295b70bac43c0a54c15c239835d0d: ending tablet copy session 05e64346b296436689045e4af52dd145-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 05e64346b296436689045e4af52dd145
I20260621 06:52:33.507705 27661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:33.510334 27859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:33.513563 27860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:33.518101 27862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:33.523211 27661 server_base.cc:1034] running on GCE node
I20260621 06:52:33.523563 27661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:33.524215 27661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:33.525416 27661 hybrid_clock.cc:648] HybridClock initialized: now 1782024753525372 us; error 40 us; skew 500 ppm
I20260621 06:52:33.525638 27661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:33 dist-test-slave-gmf2 krb5kdc[27055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024753, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:33.527781 27661 init.cc:377] Logged in from keytab as oryx/127.20.120.252@KRBTEST.COM (short username oryx)
I20260621 06:52:33.529112 27661 webserver.cc:460] Webserver started at http://127.20.120.252:36091/ using document root <none> and password file <none>
I20260621 06:52:33.529445 27661 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:33.529551 27661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:33.532582 27661 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260621 06:52:33.534653 27869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:33.535039 27661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260621 06:52:33.535153 27661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/wal
uuid: "05e64346b296436689045e4af52dd145"
format_stamp: "Formatted at 2026-06-21 06:52:33 on dist-test-slave-gmf2"
server_key: "daa0aef2a77b8f6286a1ae84952c9705"
server_key_iv: "93d59288e97754fdfa4b424815a2a406"
server_key_version: "encryptionkey@0"
I20260621 06:52:33.535333 27661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:33.548993 27661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:33.552839 27661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:33.586261 27661 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.252:34201
I20260621 06:52:33.586314 27920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.252:34201 every 8 connection(s)
I20260621 06:52:33.587172 27661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
I20260621 06:52:33.592010 27921 sys_catalog.cc:263] Verifying existing consensus state
I20260621 06:52:33.596103 27921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: Bootstrap starting.
I20260621 06:52:33.614339 27921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 06:52:33.614916 27921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: Bootstrap complete.
I20260621 06:52:33.616291 27921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } }
I20260621 06:52:33.616659 27921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 05e64346b296436689045e4af52dd145, State: Initialized, Role: LEARNER
I20260621 06:52:33.616991 27921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } }
I20260621 06:52:33.617957 27924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } } }
I20260621 06:52:33.618291 27921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:33.618464 27924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: This master's current role is: LEARNER
I20260621 06:52:33.628129 27935 catalog_manager.cc:1259] Loaded cluster ID: 2660fb547c4648959bd33249505d04a5
I20260621 06:52:33.628226 27935 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: loading cluster ID for follower catalog manager: success
I20260621 06:52:33.631546 27935 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: acquiring CA information for follower catalog manager: success
I20260621 06:52:33.633836 27935 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 06:52:33.831316 27936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.824595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44484 (local address 127.20.120.252:34201)
0621 06:52:33.825123 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.825127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.825144 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:52:33.825204 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.825207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.825275 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.825380 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:33.825875 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.826852 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.829470 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.830368 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.830520 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.830649 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.830984 (+ 335us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.831073 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":300,"threads_started":1}
I20260621 06:52:33.831441 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.824420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:34201 (local address 127.0.0.1:44484)
0621 06:52:33.824641 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.824674 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.824828 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.825438 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.825445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:33.825477 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:33.825718 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.825730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.827005 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.827011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.829320 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.829338 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.830536 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.830543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.830814 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.830853 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 06:52:33.830970 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 06:52:33.831121 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 06:52:33.832356 27896 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260621 06:52:33.832928 27924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e295b70bac43c0a54c15c239835d0d. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } } }
I20260621 06:52:33.833201 27924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: This master's current role is: LEARNER
W20260621 06:52:33.849026 27858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:33.849364 27858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:33.849534 27858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:33.856969 27858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:33.857141 27858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:33.857338 27858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:52:33.868207 27858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab
--principal=oryx/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:40259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=42851
--tserver_master_addrs=127.20.120.254:45135,127.20.120.253:34695,127.20.120.252:34201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:33.869606 27858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:33.871018 27858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:33.885022 27944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:33.885061 27942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:33.885103 27941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:33.885788 27858 server_base.cc:1034] running on GCE node
I20260621 06:52:33.886463 27858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:33.887420 27858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:33.888607 27858 hybrid_clock.cc:648] HybridClock initialized: now 1782024753888566 us; error 39 us; skew 500 ppm
I20260621 06:52:33.888906 27858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:33 dist-test-slave-gmf2 krb5kdc[27055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024753, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:33.891856 27858 init.cc:377] Logged in from keytab as oryx/127.20.120.193@KRBTEST.COM (short username oryx)
I20260621 06:52:33.893175 27858 webserver.cc:460] Webserver started at http://127.20.120.193:42851/ using document root <none> and password file <none>
I20260621 06:52:33.893837 27858 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:33.894014 27858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:33.899374 27858 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:52:33.902006 27951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:33.903255 27858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:52:33.903384 27858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "dea92d7a226f4d5a8be376c02695cd4a"
format_stamp: "Formatted at 2026-06-21 06:52:30 on dist-test-slave-gmf2"
server_key: "2befbcc7f47f4d2127a847a7491a0880"
server_key_iv: "e7369706f252d9ee3aca9aae64672103"
server_key_version: "encryptionkey@0"
I20260621 06:52:33.904294 27858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:33.924948 27858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:33.928555 27858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:33.928774 27858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:33.929661 27858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:33.931003 27858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:33.931109 27858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:33.931231 27858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:33.931315 27858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:33.966073 27858 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:40259
I20260621 06:52:33.966137 28063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:40259 every 8 connection(s)
I20260621 06:52:33.967734 27858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:52:33.971350 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 27858
I20260621 06:52:33.971702 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 27295
Jun 21 06:52:33 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024753, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:33.996838 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.970883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:35383 (local address 127.20.120.254:45135)
0621 06:52:33.971080 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.971089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.975302 (+ 4213us) server_negotiation.cc:407] Connection header received
0621 06:52:33.979139 (+ 3837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.979148 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.979257 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.979342 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.982328 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.983425 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.985472 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.985863 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.986090 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.989672 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:33.989677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:33.989680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:33.989710 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:33.993109 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.994143 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.994150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.994156 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.994318 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:33.994853 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:33.994858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:33.994862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:33.995172 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:33.995302 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:33.996260 (+ 958us) server_negotiation.cc:299] Negotiation successful
0621 06:52:33.996492 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 06:52:33.999456 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.971235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.193:35383)
0621 06:52:33.975106 (+ 3871us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.975177 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.978736 (+ 3559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.980009 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.980025 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:33.981017 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:33.982083 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.982116 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.983578 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.983582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.985332 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.985343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.986179 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.986186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.986494 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:33.987198 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:33.987220 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:33.989442 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:33.993299 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.993307 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.993325 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.993998 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.994472 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:33.994476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:33.994479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:33.994707 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:33.995394 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:33.995402 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:33.995779 (+ 377us) client_negotiation.cc:769] Sending connection context
0621 06:52:33.997331 (+ 1552us) client_negotiation.cc:240] Negotiation successful
0621 06:52:33.997933 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3460,"thread_start_us":3233,"threads_started":1}
I20260621 06:52:34.001297 28065 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45135
I20260621 06:52:34.001633 28065 heartbeater.cc:464] Registering TS with master...
Jun 21 06:52:34 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024753, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.253@KRBTEST.COM
I20260621 06:52:34.002642 28065 heartbeater.cc:511] Master 127.20.120.254:45135 requested a full tablet report, sending...
Jun 21 06:52:34 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.120.194@KRBTEST.COM
I20260621 06:52:34.004690 27100 ts_manager.cc:194] Re-registered known tserver with Master: dea92d7a226f4d5a8be376c02695cd4a (127.20.120.193:40259)
I20260621 06:52:34.005847 27100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:35383
Jun 21 06:52:34 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024753, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.252@KRBTEST.COM
I20260621 06:52:34.010795 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.970613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:54787 (local address 127.20.120.253:34695)
0621 06:52:33.983085 (+ 12472us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.983091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.991958 (+ 8867us) server_negotiation.cc:407] Connection header received
0621 06:52:33.992027 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.992032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.992116 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.992372 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.994055 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.995500 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.998082 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.998838 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.998923 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:34.003616 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:34.003623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:34.003628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:34.003672 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:34.007659 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.008570 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.008578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.008584 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.008727 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.009225 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.009232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.009286 (+ 54us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.009630 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:34.009789 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:34.010225 (+ 436us) server_negotiation.cc:299] Negotiation successful
0621 06:52:34.010434 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12349,"thread_start_us":12476,"threads_started":1}
I20260621 06:52:34.010856 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.971850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34695 (local address 127.20.120.193:54787)
0621 06:52:33.991699 (+ 19849us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.991724 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.991835 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.992614 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.992621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:33.993407 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:33.993813 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.993829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.995726 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.995734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.997869 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:33.997887 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.999185 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.999193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:33.999374 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:34.000369 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:34.000406 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:34.003416 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:34.007815 (+ 4399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:34.007823 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:34.007829 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:34.008350 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:34.008924 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:34.008929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:34.008933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:34.009093 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:34.009772 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:34.009777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:34.009955 (+ 178us) client_negotiation.cc:769] Sending connection context
0621 06:52:34.010109 (+ 154us) client_negotiation.cc:240] Negotiation successful
0621 06:52:34.010451 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19718,"thread_start_us":19554,"threads_started":1}
I20260621 06:52:34.011839 28066 heartbeater.cc:347] Connected to a master server at 127.20.120.253:34695
I20260621 06:52:34.011998 28066 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:34.012425 28066 heartbeater.cc:511] Master 127.20.120.253:34695 requested a full tablet report, sending...
I20260621 06:52:34.013309 28072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.973023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.20.120.194:43333)
0621 06:52:33.990723 (+ 17700us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.990733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.991532 (+ 799us) server_negotiation.cc:407] Connection header received
0621 06:52:33.991615 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.991624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.991930 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.992088 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:33.994283 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.995809 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.997606 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:33.998420 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:33.998549 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:34.005280 (+ 6731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:34.005295 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:34.005323 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:34.005363 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:34.010180 (+ 4817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.010994 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.011002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.011005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.011161 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.011523 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.011527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.011530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.011895 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:34.012232 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:34.012569 (+ 337us) server_negotiation.cc:299] Negotiation successful
0621 06:52:34.012803 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17516,"thread_start_us":17360,"threads_started":1}
I20260621 06:52:34.014101 27697 ts_manager.cc:194] Registered new tserver with Master: dea92d7a226f4d5a8be376c02695cd4a (127.20.120.193:40259)
I20260621 06:52:34.014492 27418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:34.016085 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.992282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:34201 (local address 127.20.120.193:42383)
0621 06:52:33.998806 (+ 6524us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:33.998825 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:33.999153 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:33.999676 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:33.999680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:34.000713 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:34.001065 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:34.001079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.002515 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.002519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:34.004122 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:34.004137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.005343 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.005351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:34.005511 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:34.006037 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:34.006057 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:34.009197 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:34.013225 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:34.013229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:34.013233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:34.013663 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:34.014099 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:34.014103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:34.014106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:34.014312 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:34.015030 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:34.015037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:34.015241 (+ 204us) client_negotiation.cc:769] Sending connection context
0621 06:52:34.015421 (+ 180us) client_negotiation.cc:240] Negotiation successful
0621 06:52:34.015705 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6401,"thread_start_us":6650,"threads_started":1}
I20260621 06:52:34.016426 27936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:33.992090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:42383 (local address 127.20.120.252:34201)
0621 06:52:33.992242 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:33.992248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:33.998917 (+ 6669us) server_negotiation.cc:407] Connection header received
0621 06:52:33.999306 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:33.999312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:33.999388 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:33.999458 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:34.001229 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.002312 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.004287 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.004979 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.005081 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:34.009445 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:34.009450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:34.009453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:34.009484 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:34.013085 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.013790 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.013794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.013798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.013963 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.014435 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.014439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.014442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.014824 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:34.014944 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:34.015506 (+ 562us) server_negotiation.cc:299] Negotiation successful
0621 06:52:34.015766 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 06:52:34.017261 28064 heartbeater.cc:347] Connected to a master server at 127.20.120.252:34201
I20260621 06:52:34.017429 28064 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:34.017812 28064 heartbeater.cc:511] Master 127.20.120.252:34201 requested a full tablet report, sending...
I20260621 06:52:34.019604 27886 ts_manager.cc:194] Registered new tserver with Master: dea92d7a226f4d5a8be376c02695cd4a (127.20.120.193:40259)
I20260621 06:52:34.101293 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 27295
I20260621 06:52:34.117498 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:43333
--local_ip_for_outbound_sockets=127.20.120.194
--tserver_master_addrs=127.20.120.254:45135
--webserver_port=40245
--webserver_interface=127.20.120.194
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab
--principal=oryx/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.120.254:45135,127.20.120.253:34695
--tserver_master_addrs=127.20.120.254:45135,127.20.120.253:34695,127.20.120.252:34201 with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:52:34.272233 27844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: NON_VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: true } } }
I20260621 06:52:34.272691 27844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:34.274477 28076 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d: attempting to promote NON_VOTER 05e64346b296436689045e4af52dd145 to VOTER
I20260621 06:52:34.275043 28076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } }
I20260621 06:52:34.276850 27707 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 FOLLOWER]: Refusing update from remote peer 85e295b70bac43c0a54c15c239835d0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260621 06:52:34.276908 27896 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 LEARNER]: Refusing update from remote peer 85e295b70bac43c0a54c15c239835d0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260621 06:52:34.277437 27844 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260621 06:52:34.277988 28077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260621 06:52:34.280372 27844 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05e64346b296436689045e4af52dd145 (127.20.120.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } } }
I20260621 06:52:34.280992 27707 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05e64346b296436689045e4af52dd145 (127.20.120.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } } }
I20260621 06:52:34.284444 27844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } } }
I20260621 06:52:34.285006 27844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:34.285233 27844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } } }
I20260621 06:52:34.285663 27844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:34.287892 27844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } } }
I20260621 06:52:34.288209 27844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:34.288987 27896 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 05e64346b296436689045e4af52dd145 (127.20.120.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } } }
I20260621 06:52:34.292732 28080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } } }
I20260621 06:52:34.293105 28080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:52:34.293737 27924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85e295b70bac43c0a54c15c239835d0d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } } }
I20260621 06:52:34.294086 27924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: This master's current role is: FOLLOWER
W20260621 06:52:34.448563 28075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:34.448830 28075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:34.448985 28075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:34.456246 28075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:34.456359 28075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:34.456552 28075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:52:34.466107 28075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab
--principal=oryx/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:43333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=40245
--tserver_master_addrs=127.20.120.254:45135,127.20.120.253:34695,127.20.120.252:34201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:34.467437 28075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:34.468768 28075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:34.481792 28085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:34.481768 28088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:34.482242 28086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:34.482492 28075 server_base.cc:1034] running on GCE node
I20260621 06:52:34.483096 28075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:34.483978 28075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:34.485191 28075 hybrid_clock.cc:648] HybridClock initialized: now 1782024754485154 us; error 47 us; skew 500 ppm
I20260621 06:52:34.485509 28075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:34 dist-test-slave-gmf2 krb5kdc[27055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024754, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:34.488562 28075 init.cc:377] Logged in from keytab as oryx/127.20.120.194@KRBTEST.COM (short username oryx)
I20260621 06:52:34.490015 28075 webserver.cc:460] Webserver started at http://127.20.120.194:40245/ using document root <none> and password file <none>
I20260621 06:52:34.490687 28075 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:34.490835 28075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:34.495971 28075 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 06:52:34.498617 28095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:34.499944 28075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:34.500133 28075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "0b8f4bce11874dd39985773f5e4a5e49"
format_stamp: "Formatted at 2026-06-21 06:52:31 on dist-test-slave-gmf2"
server_key: "01f3ab007ded71eef63fa13e0e6dd2e0"
server_key_iv: "fd89af56dfd774cbdd4b657addcd536c"
server_key_version: "encryptionkey@0"
I20260621 06:52:34.501222 28075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:34.520860 28075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:34.524848 28075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:34.525115 28075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:34.526103 28075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:34.527390 28075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:34.527518 28075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:34.527637 28075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:34.527717 28075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:34.559659 28075 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:43333
I20260621 06:52:34.559751 28207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:43333 every 8 connection(s)
I20260621 06:52:34.561048 28075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:52:34.568499 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 28075
I20260621 06:52:34.568794 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 27428
Jun 21 06:52:34 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024754, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.252@KRBTEST.COM
Jun 21 06:52:34 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024754, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.253@KRBTEST.COM
Jun 21 06:52:34 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024754, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
Jun 21 06:52:34 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.120.195@KRBTEST.COM
I20260621 06:52:34.588161 28213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:34.564006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:52095 (local address 127.20.120.252:34201)
0621 06:52:34.564864 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:34.564871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:34.565672 (+ 801us) server_negotiation.cc:407] Connection header received
0621 06:52:34.567999 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:34.568006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:34.568095 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:34.568220 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:34.570862 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.571597 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.573856 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.574523 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.574623 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:34.580074 (+ 5451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:34.580080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:34.580085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:34.580150 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:34.584181 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.585073 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.585079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.585085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.585263 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.585850 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.585856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.585860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.586252 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:34.586395 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:34.587490 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0621 06:52:34.587753 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":623,"threads_started":1}
I20260621 06:52:34.589483 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:34.566068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34695 (local address 127.20.120.194:50249)
0621 06:52:34.566788 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:34.566808 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:34.567755 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:34.568842 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:34.568847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:34.569533 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:34.570458 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:34.570484 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.572352 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.572357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:34.573875 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:34.573887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.575018 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.575023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:34.575328 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:34.576347 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:34.576364 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:34.580237 (+ 3873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:34.584200 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:34.584209 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:34.584229 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:34.585736 (+ 1507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:34.586177 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:34.586184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:34.586190 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:34.586376 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:34.587100 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:34.587105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:34.587244 (+ 139us) client_negotiation.cc:769] Sending connection context
0621 06:52:34.587428 (+ 184us) client_negotiation.cc:240] Negotiation successful
0621 06:52:34.587959 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":233,"thread_start_us":363,"threads_started":1}
I20260621 06:52:34.589519 28214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:34.564605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:34201 (local address 127.20.120.194:52095)
0621 06:52:34.565498 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:34.565554 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:34.567743 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:34.568453 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:34.568461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:34.569318 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:34.570734 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:34.570742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.571744 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.571750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:34.573683 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:34.573697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.574948 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.574955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:34.575323 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:34.576204 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:34.576234 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:34.579838 (+ 3604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:34.584314 (+ 4476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:34.584318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:34.584321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:34.584923 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:34.585442 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:34.585449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:34.585456 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:34.585710 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:34.586518 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:34.586528 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:34.586983 (+ 455us) client_negotiation.cc:769] Sending connection context
0621 06:52:34.588392 (+ 1409us) client_negotiation.cc:240] Negotiation successful
0621 06:52:34.588710 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":137,"spinlock_wait_cycles":58368,"thread_start_us":386,"threads_started":1}
I20260621 06:52:34.591033 28208 heartbeater.cc:347] Connected to a master server at 127.20.120.252:34201
I20260621 06:52:34.591365 28208 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:34.591820 28217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:34.566345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:50249 (local address 127.20.120.253:34695)
0621 06:52:34.567292 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:34.567296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:34.567318 (+ 22us) server_negotiation.cc:407] Connection header received
0621 06:52:34.568558 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:34.568566 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:34.568648 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:34.568813 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:34.570650 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.572163 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.574051 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.574653 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.574775 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:34.580471 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:34.580477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:34.580482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:34.580526 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:34.584030 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.585880 (+ 1850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.585885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.585889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.586050 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.586563 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.586568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.586571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.586899 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:34.587043 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:34.591202 (+ 4159us) server_negotiation.cc:299] Negotiation successful
0621 06:52:34.591474 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":753,"threads_started":1}
I20260621 06:52:34.592278 28208 heartbeater.cc:511] Master 127.20.120.252:34201 requested a full tablet report, sending...
I20260621 06:52:34.594198 27886 ts_manager.cc:194] Registered new tserver with Master: 0b8f4bce11874dd39985773f5e4a5e49 (127.20.120.194:43333)
I20260621 06:52:34.595024 28210 heartbeater.cc:347] Connected to a master server at 127.20.120.253:34695
I20260621 06:52:34.595140 28210 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:34.595502 28210 heartbeater.cc:511] Master 127.20.120.253:34695 requested a full tablet report, sending...
I20260621 06:52:34.596577 27697 ts_manager.cc:194] Registered new tserver with Master: 0b8f4bce11874dd39985773f5e4a5e49 (127.20.120.194:43333)
I20260621 06:52:34.598224 28219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:34.570159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.20.120.195:43025)
0621 06:52:34.574469 (+ 4310us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:34.574480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:34.574510 (+ 30us) server_negotiation.cc:407] Connection header received
0621 06:52:34.574584 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:34.574595 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:34.574838 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:34.577488 (+ 2650us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:34.578693 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.580094 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.581877 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.583216 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.583351 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:34.589432 (+ 6081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:34.589449 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:34.589469 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:34.589521 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:34.594305 (+ 4784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.595215 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.595223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.595227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.595382 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.595818 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.595824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.595827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.596238 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:34.596494 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:34.597624 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0621 06:52:34.597802 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4136,"thread_start_us":3999,"threads_started":1}
I20260621 06:52:34.599022 28215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:34.565013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.194:60897)
0621 06:52:34.565706 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:34.565723 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:34.567755 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:34.569026 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:34.569031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:34.569701 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:34.570458 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:34.570484 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.574435 (+ 3951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.574440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:34.577448 (+ 3008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:34.577461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.578349 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.578356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:34.578570 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:34.579141 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:34.579162 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:34.584220 (+ 5058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:34.596237 (+ 12017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:34.596243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:34.596247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:34.596724 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:34.597180 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:34.597186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:34.597191 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:34.597353 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:34.598053 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:34.598059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:34.598220 (+ 161us) client_negotiation.cc:769] Sending connection context
0621 06:52:34.598362 (+ 142us) client_negotiation.cc:240] Negotiation successful
0621 06:52:34.598657 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":128,"thread_start_us":405,"threads_started":1}
I20260621 06:52:34.599524 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:34.563897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:60897 (local address 127.20.120.254:45135)
0621 06:52:34.564494 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:34.564504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:34.565800 (+ 1296us) server_negotiation.cc:407] Connection header received
0621 06:52:34.568136 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:34.568143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:34.568225 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:34.568359 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:34.572407 (+ 4048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.573831 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.577593 (+ 3762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:34.578157 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:34.578295 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:34.584410 (+ 6115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:34.584417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:34.584422 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:34.584462 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:34.596073 (+ 11611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.596872 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.596879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.596886 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.597035 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:34.597525 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:34.597530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:34.597533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:34.597896 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:34.598049 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:34.598725 (+ 676us) server_negotiation.cc:299] Negotiation successful
0621 06:52:34.598938 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":305,"threads_started":1}
I20260621 06:52:34.599538 27551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:34.600400 28209 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45135
I20260621 06:52:34.600549 28209 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:34.601096 28209 heartbeater.cc:511] Master 127.20.120.254:45135 requested a full tablet report, sending...
I20260621 06:52:34.602324 27100 ts_manager.cc:194] Re-registered known tserver with Master: 0b8f4bce11874dd39985773f5e4a5e49 (127.20.120.194:43333)
I20260621 06:52:34.603796 27100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:60897
I20260621 06:52:34.687155 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 27428
I20260621 06:52:34.703037 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:43025
--local_ip_for_outbound_sockets=127.20.120.195
--tserver_master_addrs=127.20.120.254:45135
--webserver_port=44367
--webserver_interface=127.20.120.195
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab
--principal=oryx/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.120.254:45135,127.20.120.253:34695
--tserver_master_addrs=127.20.120.254:45135,127.20.120.253:34695,127.20.120.252:34201 with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:52:35.009495 28065 heartbeater.cc:503] Master 127.20.120.254:45135 was elected leader, sending a full tablet report...
W20260621 06:52:35.032207 28221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:35.032480 28221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:35.032572 28221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:35.039817 28221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:35.039927 28221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:35.040047 28221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:52:35.050655 28221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab
--principal=oryx/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:43025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=44367
--tserver_master_addrs=127.20.120.254:45135,127.20.120.253:34695,127.20.120.252:34201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:35.051993 28221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:35.053272 28221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:35.066228 28228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:35.066267 28229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:35.066547 28231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:35.067138 28221 server_base.cc:1034] running on GCE node
I20260621 06:52:35.067721 28221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:35.068625 28221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:35.069811 28221 hybrid_clock.cc:648] HybridClock initialized: now 1782024755069748 us; error 62 us; skew 500 ppm
I20260621 06:52:35.070140 28221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:35 dist-test-slave-gmf2 krb5kdc[27055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024755, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:35.073104 28221 init.cc:377] Logged in from keytab as oryx/127.20.120.195@KRBTEST.COM (short username oryx)
I20260621 06:52:35.074559 28221 webserver.cc:460] Webserver started at http://127.20.120.195:44367/ using document root <none> and password file <none>
I20260621 06:52:35.075209 28221 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:35.075357 28221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:35.081081 28221 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260621 06:52:35.083803 28238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:35.085120 28221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260621 06:52:35.085281 28221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "f7c8fde0dfaf48099d75ed7dec484618"
format_stamp: "Formatted at 2026-06-21 06:52:31 on dist-test-slave-gmf2"
server_key: "7ee1795085f55da5cb72bfe0ce8a0b78"
server_key_iv: "a95aa4d20f9b93a5dc7c1eaaadffc7c1"
server_key_version: "encryptionkey@0"
I20260621 06:52:35.086351 28221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:35.128844 28221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:35.132864 28221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:35.133090 28221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:35.134021 28221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:35.135281 28221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:35.135372 28221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:35.135526 28221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:35.135591 28221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:35.167055 28221 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:43025
I20260621 06:52:35.167138 28350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:43025 every 8 connection(s)
I20260621 06:52:35.168444 28221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:52:35.173228 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 28221
Jun 21 06:52:35 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024755, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.253@KRBTEST.COM
Jun 21 06:52:35 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024755, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
Jun 21 06:52:35 dist-test-slave-gmf2 krb5kdc[27055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024755, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.252@KRBTEST.COM
I20260621 06:52:35.211321 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.171899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34695 (local address 127.20.120.195:43117)
0621 06:52:35.178320 (+ 6421us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:35.178381 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:35.181715 (+ 3334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:35.183002 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:35.183017 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:35.183970 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:35.184989 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.185018 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.187053 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.187056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.188518 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.188529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.191177 (+ 2648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.191190 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.191523 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.192402 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:35.192434 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:35.195657 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:35.199859 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:35.199869 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:35.199887 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:35.200568 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:35.204391 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:35.204396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:35.204400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:35.204660 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:35.207677 (+ 3017us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:35.207686 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:35.208351 (+ 665us) client_negotiation.cc:769] Sending connection context
0621 06:52:35.208802 (+ 451us) client_negotiation.cc:240] Negotiation successful
0621 06:52:35.209533 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":17,"thread_start_us":585,"threads_started":1}
I20260621 06:52:35.212651 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.171137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:43117 (local address 127.20.120.253:34695)
0621 06:52:35.177620 (+ 6483us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.177632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.182168 (+ 4536us) server_negotiation.cc:407] Connection header received
0621 06:52:35.182255 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.182263 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.182384 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.182505 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:35.185954 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.186901 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.188654 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.189310 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.189443 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.195905 (+ 6462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:35.195911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:35.195916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:35.195954 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:35.199682 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.200709 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.200715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.200718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.200883 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.204807 (+ 3924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.204814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.204817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.205199 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:35.205371 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.211873 (+ 6502us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.212204 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6358,"thread_start_us":6234,"threads_started":1}
I20260621 06:52:35.214538 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.171085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:43479 (local address 127.20.120.254:45135)
0621 06:52:35.171564 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.171573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.185718 (+ 14145us) server_negotiation.cc:407] Connection header received
0621 06:52:35.185889 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.185895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.186010 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.186121 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:35.189704 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.190879 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.193477 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.194067 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.194198 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.201104 (+ 6906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:35.201110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:35.201114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:35.201170 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:35.205370 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.206370 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.206376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.206379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.206524 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.206955 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.206962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.206968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.207285 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:35.207420 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.213973 (+ 6553us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.214189 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":219,"threads_started":1}
I20260621 06:52:35.214632 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.174023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:46677 (local address 127.20.120.252:34201)
0621 06:52:35.190280 (+ 16257us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.190286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.190878 (+ 592us) server_negotiation.cc:407] Connection header received
0621 06:52:35.190944 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.190950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.191046 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.191146 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:35.192479 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.193567 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.195911 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.196542 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.196669 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.201442 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:35.201448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:35.201453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:35.201492 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:35.207354 (+ 5862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.211470 (+ 4116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.211478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.211483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.211617 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.212678 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.212684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.212687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.213024 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:35.213182 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.214052 (+ 870us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.214261 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16133,"thread_start_us":16031,"threads_started":1}
I20260621 06:52:35.215219 28353 heartbeater.cc:347] Connected to a master server at 127.20.120.253:34695
I20260621 06:52:35.215639 28353 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:35.216156 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.173613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:34201 (local address 127.20.120.195:46677)
0621 06:52:35.190661 (+ 17048us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:35.190678 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:35.190779 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:35.191331 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:35.191337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:35.191950 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:35.192295 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.192321 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.193717 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.193726 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.195755 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.195775 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.196735 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.196743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.196962 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.197585 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:35.197616 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:35.201203 (+ 3587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:35.210801 (+ 9598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:35.210805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:35.210809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:35.211300 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:35.212388 (+ 1088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:35.212394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:35.212398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:35.212545 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:35.213512 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:35.213517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:35.213658 (+ 141us) client_negotiation.cc:769] Sending connection context
0621 06:52:35.213787 (+ 129us) client_negotiation.cc:240] Negotiation successful
0621 06:52:35.215541 (+ 1754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16947,"thread_start_us":16826,"threads_started":1}
I20260621 06:52:35.216585 28353 heartbeater.cc:511] Master 127.20.120.253:34695 requested a full tablet report, sending...
I20260621 06:52:35.217038 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.172619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.195:43479)
0621 06:52:35.185612 (+ 12993us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:35.185626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:35.185755 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:35.188702 (+ 2947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:35.188707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:35.189271 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:35.189563 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.189574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.191028 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.191037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.193304 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.193323 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.196027 (+ 2704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.196036 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.196261 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.196940 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:35.196965 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:35.200896 (+ 3931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:35.205574 (+ 4678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:35.205582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:35.205588 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:35.206231 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:35.206666 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:35.206671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:35.206675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:35.206825 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:35.207677 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:35.207686 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:35.213678 (+ 5992us) client_negotiation.cc:769] Sending connection context
0621 06:52:35.215338 (+ 1660us) client_negotiation.cc:240] Negotiation successful
0621 06:52:35.216677 (+ 1339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12868,"mutex_wait_us":1104,"thread_start_us":12624,"threads_started":1}
I20260621 06:52:35.217075 28351 heartbeater.cc:347] Connected to a master server at 127.20.120.252:34201
I20260621 06:52:35.217197 28351 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:35.217631 28351 heartbeater.cc:511] Master 127.20.120.252:34201 requested a full tablet report, sending...
I20260621 06:52:35.218410 28352 heartbeater.cc:347] Connected to a master server at 127.20.120.254:45135
I20260621 06:52:35.218474 27697 ts_manager.cc:194] Registered new tserver with Master: f7c8fde0dfaf48099d75ed7dec484618 (127.20.120.195:43025)
I20260621 06:52:35.218546 28352 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:35.219038 28352 heartbeater.cc:511] Master 127.20.120.254:45135 requested a full tablet report, sending...
I20260621 06:52:35.219914 27886 ts_manager.cc:194] Registered new tserver with Master: f7c8fde0dfaf48099d75ed7dec484618 (127.20.120.195:43025)
I20260621 06:52:35.220168 27100 ts_manager.cc:194] Re-registered known tserver with Master: f7c8fde0dfaf48099d75ed7dec484618 (127.20.120.195:43025)
I20260621 06:52:35.221045 27100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:43479
I20260621 06:52:35.238821 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.201162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.20.120.254:45135)
0621 06:52:35.215287 (+ 14125us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.215295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.220004 (+ 4709us) server_negotiation.cc:407] Connection header received
0621 06:52:35.220300 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.220308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.220453 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.220539 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:35.224251 (+ 3712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.225600 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.227406 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.227802 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.227895 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.231801 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:35.231807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:35.231812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:35.231850 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:35.235266 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.236208 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.236215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.236220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.236359 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.236945 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.236952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.236957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.237324 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:35.237568 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.238195 (+ 627us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.238417 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":14000,"thread_start_us":22446,"threads_started":1}
I20260621 06:52:35.241482 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.199699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.20.120.253:34695)
0621 06:52:35.213216 (+ 13517us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.213224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.226239 (+ 13015us) server_negotiation.cc:407] Connection header received
0621 06:52:35.226420 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.226426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.226516 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.226607 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:35.229260 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.229991 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.231452 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.232022 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.232152 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.235193 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:35.235198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:35.235203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:35.235241 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:35.238619 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.239501 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.239507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.239511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.239649 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.240037 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.240043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.240047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.240326 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:35.240585 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.240931 (+ 346us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.241154 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13409,"thread_start_us":16266,"threads_started":1}
I20260621 06:52:35.244407 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.208553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.20.120.252:34201)
0621 06:52:35.214847 (+ 6294us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.214852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.226890 (+ 12038us) server_negotiation.cc:407] Connection header received
0621 06:52:35.227088 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.227093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.227206 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.227334 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:35.231343 (+ 4009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.232749 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.234805 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.235481 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.235607 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.238341 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:35.238345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:35.238349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:35.238377 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:35.241587 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.242427 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.242435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.242438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.242575 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.242974 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.242981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.242984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.243316 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:35.243547 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.243863 (+ 316us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.244100 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6205,"thread_start_us":18655,"threads_started":1}
I20260621 06:52:35.243698 27100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:52:35.250982 27100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:52:35.290019 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.268704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.20.120.193:40259)
0621 06:52:35.275854 (+ 7150us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.275867 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.275911 (+ 44us) server_negotiation.cc:407] Connection header received
0621 06:52:35.276078 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.276090 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.276472 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.276765 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:35.279022 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.283568 (+ 4546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.287434 (+ 3866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.288905 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.289087 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.289275 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.289486 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.289655 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6885,"thread_start_us":262,"threads_started":1}
I20260621 06:52:35.290071 28378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.268336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:40259 (local address 127.0.0.1:46144)
0621 06:52:35.268901 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:35.268926 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:35.269056 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:35.276748 (+ 7692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:35.276753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:35.276783 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:35.277129 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.277145 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.283729 (+ 6584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.283739 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.286707 (+ 2968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.286723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.289092 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.289098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.289280 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.289328 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 06:52:35.289430 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 06:52:35.289640 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":44,"thread_start_us":382,"threads_started":1}
I20260621 06:52:35.294517 28381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.269444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.20.120.194:43333)
0621 06:52:35.284842 (+ 15398us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.284855 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.286578 (+ 1723us) server_negotiation.cc:407] Connection header received
0621 06:52:35.286829 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.286840 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.287212 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.287450 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:35.287994 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.288925 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.292236 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.293138 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.293304 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.293480 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.294079 (+ 599us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.294250 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15149,"thread_start_us":15815,"threads_started":1}
I20260621 06:52:35.294111 27999 tablet_service.cc:1467] Processing CreateTablet for tablet 8c32a6d2f49a48e5ba612f3288839e4e (DEFAULT_TABLE table=test-table [id=ab009ad7c3784e19975cb85fdca83b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:35.295440 27999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c32a6d2f49a48e5ba612f3288839e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:35.296133 28382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.269214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:43333 (local address 127.0.0.1:57112)
0621 06:52:35.286086 (+ 16872us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:35.286460 (+ 374us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:35.286653 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:35.287452 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:35.287458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:35.287491 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:35.287832 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.287843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.289074 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.289079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.292097 (+ 3018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.292108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.293610 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.293616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.293788 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.293836 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 06:52:35.293957 (+ 121us) client_negotiation.cc:240] Negotiation successful
0621 06:52:35.295294 (+ 1337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16730,"mutex_wait_us":2005,"thread_start_us":20077,"threads_started":1}
I20260621 06:52:35.298404 28380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.269645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:43025 (local address 127.0.0.1:41820)
0621 06:52:35.286748 (+ 17103us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:35.286766 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:35.289487 (+ 2721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:35.290415 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:35.290421 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:35.290445 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:35.290850 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.290861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.293151 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.293155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.295127 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.295137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.297579 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.297585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.297773 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.297805 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 06:52:35.297887 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 06:52:35.298032 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16975,"mutex_wait_us":17,"thread_start_us":17191,"threads_started":1}
I20260621 06:52:35.300096 28143 tablet_service.cc:1467] Processing CreateTablet for tablet 8c32a6d2f49a48e5ba612f3288839e4e (DEFAULT_TABLE table=test-table [id=ab009ad7c3784e19975cb85fdca83b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:35.301450 28143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c32a6d2f49a48e5ba612f3288839e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:35.302438 28383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.269956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.20.120.195:43025)
0621 06:52:35.286243 (+ 16287us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.286250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.286875 (+ 625us) server_negotiation.cc:407] Connection header received
0621 06:52:35.289665 (+ 2790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.289674 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.289990 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.290217 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:35.291479 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.293002 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.295261 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.296814 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.296965 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.297119 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.301971 (+ 4852us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.302126 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15836,"thread_start_us":15986,"threads_started":1}
I20260621 06:52:35.305898 28286 tablet_service.cc:1467] Processing CreateTablet for tablet 8c32a6d2f49a48e5ba612f3288839e4e (DEFAULT_TABLE table=test-table [id=ab009ad7c3784e19975cb85fdca83b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:35.307873 28286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c32a6d2f49a48e5ba612f3288839e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:35.319133 28384 tablet_bootstrap.cc:492] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a: Bootstrap starting.
I20260621 06:52:35.321983 28386 tablet_bootstrap.cc:492] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618: Bootstrap starting.
I20260621 06:52:35.324895 28384 tablet_bootstrap.cc:654] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:35.325083 28386 tablet_bootstrap.cc:654] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:35.326117 28386 log.cc:824] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:35.326444 28385 tablet_bootstrap.cc:492] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49: Bootstrap starting.
I20260621 06:52:35.327137 28384 log.cc:824] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:35.330040 28386 tablet_bootstrap.cc:492] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618: No bootstrap required, opened a new log
I20260621 06:52:35.330084 28385 tablet_bootstrap.cc:654] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:35.330315 28386 ts_tablet_manager.cc:1397] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260621 06:52:35.330601 28384 tablet_bootstrap.cc:492] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a: No bootstrap required, opened a new log
I20260621 06:52:35.330860 28384 ts_tablet_manager.cc:1397] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260621 06:52:35.331269 28385 log.cc:824] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:35.336884 28385 tablet_bootstrap.cc:492] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49: No bootstrap required, opened a new log
I20260621 06:52:35.337133 28385 ts_tablet_manager.cc:1397] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260621 06:52:35.338570 28386 raft_consensus.cc:348] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:35.338922 28386 raft_consensus.cc:374] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:35.339018 28386 raft_consensus.cc:729] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c8fde0dfaf48099d75ed7dec484618, State: Initialized, Role: FOLLOWER
I20260621 06:52:35.339592 28386 consensus_queue.cc:260] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:35.341881 28384 raft_consensus.cc:348] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:35.342374 28384 raft_consensus.cc:374] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:35.342483 28384 raft_consensus.cc:729] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea92d7a226f4d5a8be376c02695cd4a, State: Initialized, Role: FOLLOWER
I20260621 06:52:35.343041 28384 consensus_queue.cc:260] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:35.344770 28384 ts_tablet_manager.cc:1428] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260621 06:52:35.348201 28385 raft_consensus.cc:348] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:35.348687 28385 raft_consensus.cc:374] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:35.348819 28385 raft_consensus.cc:729] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8f4bce11874dd39985773f5e4a5e49, State: Initialized, Role: FOLLOWER
I20260621 06:52:35.349567 28385 consensus_queue.cc:260] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:35.351457 28386 ts_tablet_manager.cc:1428] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618: Time spent starting tablet: real 0.021s user 0.010s sys 0.011s
I20260621 06:52:35.351751 28352 heartbeater.cc:503] Master 127.20.120.254:45135 was elected leader, sending a full tablet report...
I20260621 06:52:35.355921 28385 ts_tablet_manager.cc:1428] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49: Time spent starting tablet: real 0.019s user 0.013s sys 0.002s
I20260621 06:52:35.356153 28209 heartbeater.cc:503] Master 127.20.120.254:45135 was elected leader, sending a full tablet report...
W20260621 06:52:35.427129 28354 tablet.cc:2369] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:52:35.477669 28391 raft_consensus.cc:482] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:52:35.478044 28391 raft_consensus.cc:504] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:35.479619 28391 leader_election.cc:288] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b8f4bce11874dd39985773f5e4a5e49 (127.20.120.194:43333), f7c8fde0dfaf48099d75ed7dec484618 (127.20.120.195:43025)
W20260621 06:52:35.481182 28067 tablet.cc:2369] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:52:35.488535 28383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.480125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:34003 (local address 127.20.120.195:43025)
0621 06:52:35.480330 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.480336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.482012 (+ 1676us) server_negotiation.cc:407] Connection header received
0621 06:52:35.482281 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.482286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.482384 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.482475 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:35.483096 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.484155 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.486837 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.487633 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.487814 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.487890 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.488159 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.488291 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:52:35.488695 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.479893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:43025 (local address 127.20.120.193:34003)
0621 06:52:35.481876 (+ 1983us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:35.481897 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:35.482131 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:35.482495 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:35.482498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:35.482515 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:35.482946 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.482961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.484279 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.484284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.486704 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.486716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.487829 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.487835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.488003 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.488030 (+ 27us) client_negotiation.cc:769] Sending connection context
0621 06:52:35.488133 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 06:52:35.488259 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1837,"mutex_wait_us":244,"thread_start_us":2060,"threads_started":1}
I20260621 06:52:35.488724 28381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.480185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:48061 (local address 127.20.120.194:43333)
0621 06:52:35.480361 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.480369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.482293 (+ 1924us) server_negotiation.cc:407] Connection header received
0621 06:52:35.482437 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.482444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.482513 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.482591 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:35.483096 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.484156 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.486803 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.487830 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.488018 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.488091 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.488344 (+ 253us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.488466 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 06:52:35.488840 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.479893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:43333 (local address 127.20.120.193:48061)
0621 06:52:35.482211 (+ 2318us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:35.482224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:35.482326 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:35.482619 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:35.482624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:35.482648 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:35.482970 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.482978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.484280 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.484284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.486654 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:35.486666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.488048 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.488054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:35.488193 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.488214 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 06:52:35.488312 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 06:52:35.488434 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2204,"mutex_wait_us":170,"thread_start_us":1246,"threads_started":1}
I20260621 06:52:35.489841 28163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c32a6d2f49a48e5ba612f3288839e4e" candidate_uuid: "dea92d7a226f4d5a8be376c02695cd4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8f4bce11874dd39985773f5e4a5e49" is_pre_election: true
I20260621 06:52:35.489879 28306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c32a6d2f49a48e5ba612f3288839e4e" candidate_uuid: "dea92d7a226f4d5a8be376c02695cd4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c8fde0dfaf48099d75ed7dec484618" is_pre_election: true
I20260621 06:52:35.490438 28163 raft_consensus.cc:2454] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea92d7a226f4d5a8be376c02695cd4a in term 0.
I20260621 06:52:35.490453 28306 raft_consensus.cc:2454] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea92d7a226f4d5a8be376c02695cd4a in term 0.
I20260621 06:52:35.491097 27955 leader_election.cc:302] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dea92d7a226f4d5a8be376c02695cd4a, f7c8fde0dfaf48099d75ed7dec484618; no voters:
I20260621 06:52:35.491958 28391 raft_consensus.cc:2781] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:52:35.492118 28391 raft_consensus.cc:482] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:52:35.492221 28391 raft_consensus.cc:3037] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:35.494407 28391 raft_consensus.cc:504] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:35.495040 28391 leader_election.cc:288] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [CANDIDATE]: Term 1 election: Requested vote from peers 0b8f4bce11874dd39985773f5e4a5e49 (127.20.120.194:43333), f7c8fde0dfaf48099d75ed7dec484618 (127.20.120.195:43025)
I20260621 06:52:35.495447 28163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c32a6d2f49a48e5ba612f3288839e4e" candidate_uuid: "dea92d7a226f4d5a8be376c02695cd4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b8f4bce11874dd39985773f5e4a5e49"
I20260621 06:52:35.495515 28306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c32a6d2f49a48e5ba612f3288839e4e" candidate_uuid: "dea92d7a226f4d5a8be376c02695cd4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c8fde0dfaf48099d75ed7dec484618"
I20260621 06:52:35.495661 28163 raft_consensus.cc:3037] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:35.495723 28306 raft_consensus.cc:3037] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:35.497771 28163 raft_consensus.cc:2454] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea92d7a226f4d5a8be376c02695cd4a in term 1.
I20260621 06:52:35.497782 28306 raft_consensus.cc:2454] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea92d7a226f4d5a8be376c02695cd4a in term 1.
I20260621 06:52:35.498178 27955 leader_election.cc:302] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b8f4bce11874dd39985773f5e4a5e49, dea92d7a226f4d5a8be376c02695cd4a, f7c8fde0dfaf48099d75ed7dec484618; no voters:
I20260621 06:52:35.498526 28391 raft_consensus.cc:2781] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:35.498965 28391 raft_consensus.cc:686] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 1 LEADER]: Becoming Leader. State: Replica: dea92d7a226f4d5a8be376c02695cd4a, State: Running, Role: LEADER
I20260621 06:52:35.499645 28391 consensus_queue.cc:237] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:35.504506 27100 catalog_manager.cc:5495] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a reported cstate change: term changed from 0 to 1, leader changed from <none> to dea92d7a226f4d5a8be376c02695cd4a (127.20.120.193). New cstate: current_term: 1 leader_uuid: "dea92d7a226f4d5a8be376c02695cd4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } health_report { overall_health: UNKNOWN } } }
I20260621 06:52:35.549499 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.544803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.20.120.193:40259)
0621 06:52:35.544972 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.544976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.544992 (+ 16us) server_negotiation.cc:407] Connection header received
0621 06:52:35.545165 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.545169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.545241 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.545360 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:52:35.545634 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.546365 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.547724 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.548111 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.548193 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.548592 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:52:35.549080 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:52:35.549189 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.549245 (+ 56us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.549303 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 06:52:35.557519 28163 raft_consensus.cc:1261] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49 [term 1 FOLLOWER]: Refusing update from remote peer dea92d7a226f4d5a8be376c02695cd4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:35.557531 28306 raft_consensus.cc:1261] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618 [term 1 FOLLOWER]: Refusing update from remote peer dea92d7a226f4d5a8be376c02695cd4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:35.558527 28395 consensus_queue.cc:1035] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:35.558905 28391 consensus_queue.cc:1035] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 06:52:35.564718 28211 tablet.cc:2369] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:52:35.577368 28399 mvcc.cc:205] Tried to move back new op lower bound from 7299173398754430976 to 7299173398530019328. Current Snapshot: MvccSnapshot[applied={T|T < 7299173398754430976}]
I20260621 06:52:35.586517 28400 mvcc.cc:205] Tried to move back new op lower bound from 7299173398754430976 to 7299173398530019328. Current Snapshot: MvccSnapshot[applied={T|T < 7299173398754430976}]
I20260621 06:52:35.603470 27100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:35.603749 27100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:35.610880 27100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85e295b70bac43c0a54c15c239835d0d: Sending DeleteTablet for 3 replicas of tablet 8c32a6d2f49a48e5ba612f3288839e4e
I20260621 06:52:35.612305 27999 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c32a6d2f49a48e5ba612f3288839e4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:35 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:46144
I20260621 06:52:35.612435 28143 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c32a6d2f49a48e5ba612f3288839e4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:35 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:57112
I20260621 06:52:35.613231 28286 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c32a6d2f49a48e5ba612f3288839e4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:35 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:41820
I20260621 06:52:35.613320 28407 tablet_replica.cc:330] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a: stopping tablet replica
I20260621 06:52:35.614441 28407 raft_consensus.cc:2229] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:52:35.615432 28407 raft_consensus.cc:2258] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:35.617677 28407 ts_tablet_manager.cc:1905] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:35.627835 28407 ts_tablet_manager.cc:1918] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:35.628118 28407 log.cc:1196] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/8c32a6d2f49a48e5ba612f3288839e4e
I20260621 06:52:35.629026 28407 ts_tablet_manager.cc:1939] T 8c32a6d2f49a48e5ba612f3288839e4e P dea92d7a226f4d5a8be376c02695cd4a: Deleting consensus metadata
I20260621 06:52:35.631265 27084 catalog_manager.cc:4841] TS dea92d7a226f4d5a8be376c02695cd4a (127.20.120.193:40259): tablet 8c32a6d2f49a48e5ba612f3288839e4e (table test-table [id=ab009ad7c3784e19975cb85fdca83b41]) successfully deleted
I20260621 06:52:35.637033 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 27068
I20260621 06:52:35.637079 28409 tablet_replica.cc:330] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618: stopping tablet replica
I20260621 06:52:35.638103 27130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:35.638525 28409 raft_consensus.cc:2229] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:35.639207 28409 raft_consensus.cc:2258] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:35.641971 28408 tablet_replica.cc:330] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49: stopping tablet replica
I20260621 06:52:35.642473 28408 raft_consensus.cc:2229] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:35.643627 28409 ts_tablet_manager.cc:1905] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:35.644009 28408 raft_consensus.cc:2258] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:35.650072 28408 ts_tablet_manager.cc:1905] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:35.651657 28409 ts_tablet_manager.cc:1918] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:35.651836 28409 log.cc:1196] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/8c32a6d2f49a48e5ba612f3288839e4e
I20260621 06:52:35.652438 28409 ts_tablet_manager.cc:1939] T 8c32a6d2f49a48e5ba612f3288839e4e P f7c8fde0dfaf48099d75ed7dec484618: Deleting consensus metadata
I20260621 06:52:35.655581 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.620431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.20.120.253:34695)
0621 06:52:35.620598 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.620605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.635363 (+ 14758us) server_negotiation.cc:407] Connection header received
0621 06:52:35.635467 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.635474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.635574 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.635676 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:35.639352 (+ 3676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.640841 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.642854 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.643726 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.643861 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.648308 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:35.648314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:35.648319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:35.648361 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:35.652005 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.653472 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.653480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.653485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.653648 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.654086 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.654091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.654097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.654445 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:35.654678 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.654997 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.655210 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 06:52:35.659368 28408 ts_tablet_manager.cc:1918] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:35.659548 28408 log.cc:1196] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/8c32a6d2f49a48e5ba612f3288839e4e
I20260621 06:52:35.660105 28408 ts_tablet_manager.cc:1939] T 8c32a6d2f49a48e5ba612f3288839e4e P 0b8f4bce11874dd39985773f5e4a5e49: Deleting consensus metadata
I20260621 06:52:35.669209 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.635124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.20.120.252:34201)
0621 06:52:35.635310 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.635315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.655998 (+ 20683us) server_negotiation.cc:407] Connection header received
0621 06:52:35.656247 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.656253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.656339 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.656408 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:35.657985 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.658736 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.660161 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.660570 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.660701 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.663771 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:35.663777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:35.663780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:35.663828 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:35.666492 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.667176 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.667181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.667184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.667324 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.667695 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.667705 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.667708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.668029 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:35.668274 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.668577 (+ 303us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.668793 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260621 06:52:35.801728 27087 catalog_manager.cc:4841] TS f7c8fde0dfaf48099d75ed7dec484618 (127.20.120.195:43025): tablet 8c32a6d2f49a48e5ba612f3288839e4e (table test-table [id=ab009ad7c3784e19975cb85fdca83b41]) successfully deleted
I20260621 06:52:35.801847 27084 catalog_manager.cc:4841] TS 0b8f4bce11874dd39985773f5e4a5e49 (127.20.120.194:43333): tablet 8c32a6d2f49a48e5ba612f3288839e4e (table test-table [id=ab009ad7c3784e19975cb85fdca83b41]) successfully deleted
I20260621 06:52:35.802711 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 27068
I20260621 06:52:35.858294 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.843648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.20.120.252:34201)
0621 06:52:35.843842 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.843847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.844083 (+ 236us) server_negotiation.cc:407] Connection header received
0621 06:52:35.844239 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.844243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.844311 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.844408 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:35.845830 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.846929 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.848708 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.849119 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.849258 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.852109 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:35.852115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:35.852120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:35.852162 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:35.855479 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.856246 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.856253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.856256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.856380 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.856729 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.856735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.856739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.857029 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:35.857275 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.857660 (+ 385us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.857948 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 06:52:35.859097 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:35.842327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33530 (local address 127.20.120.253:34695)
0621 06:52:35.842573 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:35.842581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:35.843445 (+ 864us) server_negotiation.cc:407] Connection header received
0621 06:52:35.843641 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:35.843645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:35.843729 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:35.843830 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:35.845652 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.846571 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.848447 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:35.849118 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:35.849245 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:35.852933 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:35.852941 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:35.852948 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:35.852994 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:35.855630 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.856801 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.856806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.856810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.856984 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:35.857457 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:35.857462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:35.857465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:35.857841 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:35.858101 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:35.858573 (+ 472us) server_negotiation.cc:299] Negotiation successful
0621 06:52:35.858799 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 06:52:36.587946 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:36.586517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.195:47567)
0621 06:52:36.587580 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:36.587723 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":874,"threads_started":1}
I20260621 06:52:36.588379 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:36.586353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.194:46587)
0621 06:52:36.587978 (+ 1625us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:36.588126 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1491,"thread_start_us":1375,"threads_started":1}
W20260621 06:52:36.588407 28352 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:45135 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
W20260621 06:52:36.588852 28209 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:45135 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
I20260621 06:52:36.592779 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:36.591984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.193:35397)
0621 06:52:36.592406 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:36.592535 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":208,"threads_started":1}
W20260621 06:52:36.593246 28065 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:45135 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
I20260621 06:52:37.110682 28421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85e295b70bac43c0a54c15c239835d0d)
I20260621 06:52:37.111008 28421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } }
I20260621 06:52:37.111820 28422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 85e295b70bac43c0a54c15c239835d0d)
I20260621 06:52:37.112041 28422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } }
I20260621 06:52:37.112744 28421 leader_election.cc:288] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85e295b70bac43c0a54c15c239835d0d (127.20.120.254:45135), 53b9942a875049a9a62d340b0257ee7b (127.20.120.253:34695)
I20260621 06:52:37.114352 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.112924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:34654)
0621 06:52:37.113604 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.113944 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":307,"threads_started":1}
W20260621 06:52:37.115103 27684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
I20260621 06:52:37.116297 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.112848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:34646)
0621 06:52:37.115732 (+ 2884us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.115977 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2717,"thread_start_us":2560,"threads_started":1}
W20260621 06:52:37.116602 27873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
I20260621 06:52:37.122766 28422 leader_election.cc:288] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85e295b70bac43c0a54c15c239835d0d (127.20.120.254:45135), 05e64346b296436689045e4af52dd145 (127.20.120.252:34201)
I20260621 06:52:37.124238 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.123415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:34666)
0621 06:52:37.123725 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.123950 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260621 06:52:37.124485 27873 leader_election.cc:334] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 85e295b70bac43c0a54c15c239835d0d (127.20.120.254:45135): Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
I20260621 06:52:37.125229 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.117813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:34662)
0621 06:52:37.124525 (+ 6712us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.124912 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6572,"thread_start_us":6779,"threads_started":1}
W20260621 06:52:37.125423 27684 leader_election.cc:334] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 85e295b70bac43c0a54c15c239835d0d (127.20.120.254:45135): Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
I20260621 06:52:37.126539 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.113559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.20.120.253:34695)
0621 06:52:37.115186 (+ 1627us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:37.115216 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:37.116694 (+ 1478us) server_negotiation.cc:407] Connection header received
0621 06:52:37.117011 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:37.117019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:37.117130 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:37.117256 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:37.118437 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.120118 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.123547 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.125336 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.125521 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.125608 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:37.126043 (+ 435us) server_negotiation.cc:299] Negotiation successful
0621 06:52:37.126221 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1446,"spinlock_wait_cycles":1280,"thread_start_us":1220,"threads_started":1}
I20260621 06:52:37.127091 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.112961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34695 (local address 127.0.0.1:33544)
0621 06:52:37.116569 (+ 3608us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.116597 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:37.116815 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:37.117740 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:37.117748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:37.117800 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:37.118151 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.118168 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.120409 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.120417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.123325 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.123343 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.125522 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.125528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.125750 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.125833 (+ 83us) client_negotiation.cc:769] Sending connection context
0621 06:52:37.126722 (+ 889us) client_negotiation.cc:240] Negotiation successful
0621 06:52:37.126857 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3512,"thread_start_us":10415,"threads_started":1}
I20260621 06:52:37.127863 27707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "05e64346b296436689045e4af52dd145" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53b9942a875049a9a62d340b0257ee7b" is_pre_election: true
I20260621 06:52:37.128304 27707 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e64346b296436689045e4af52dd145 in term 1.
I20260621 06:52:37.128863 27871 leader_election.cc:302] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05e64346b296436689045e4af52dd145, 53b9942a875049a9a62d340b0257ee7b; no voters: 85e295b70bac43c0a54c15c239835d0d
I20260621 06:52:37.129264 28421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260621 06:52:37.129392 28421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 85e295b70bac43c0a54c15c239835d0d)
I20260621 06:52:37.129503 28421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 1 FOLLOWER]: Advancing to term 2
I20260621 06:52:37.131970 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.116726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:34201 (local address 127.0.0.1:35828)
0621 06:52:37.117269 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.117290 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:37.117486 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:37.125614 (+ 8128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:37.125619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:37.125646 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:37.126073 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.126092 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.127526 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.127530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.129634 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.129650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.131458 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.131463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.131583 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.131607 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 06:52:37.131692 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 06:52:37.131782 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404}
I20260621 06:52:37.132405 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.123660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35828 (local address 127.20.120.252:34201)
0621 06:52:37.125258 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:37.125264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:37.125287 (+ 23us) server_negotiation.cc:407] Connection header received
0621 06:52:37.125363 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:37.125367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:37.125445 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:37.126233 (+ 788us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:37.126460 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.127395 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.129823 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.131260 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.131461 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.131528 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:37.131728 (+ 200us) server_negotiation.cc:299] Negotiation successful
0621 06:52:37.131867 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478,"thread_start_us":1343,"threads_started":1}
I20260621 06:52:37.132942 28421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } }
I20260621 06:52:37.132993 27896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53b9942a875049a9a62d340b0257ee7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "05e64346b296436689045e4af52dd145" is_pre_election: true
I20260621 06:52:37.133313 27896 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 53b9942a875049a9a62d340b0257ee7b in current term 2: Already voted for candidate 05e64346b296436689045e4af52dd145 in this term.
I20260621 06:52:37.133684 28421 leader_election.cc:288] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [CANDIDATE]: Term 2 election: Requested vote from peers 85e295b70bac43c0a54c15c239835d0d (127.20.120.254:45135), 53b9942a875049a9a62d340b0257ee7b (127.20.120.253:34695)
I20260621 06:52:37.133875 27682 leader_election.cc:302] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53b9942a875049a9a62d340b0257ee7b; no voters: 05e64346b296436689045e4af52dd145, 85e295b70bac43c0a54c15c239835d0d
I20260621 06:52:37.134186 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.133714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:34680)
0621 06:52:37.133961 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.134044 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260621 06:52:37.134104 27707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "05e64346b296436689045e4af52dd145" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53b9942a875049a9a62d340b0257ee7b"
I20260621 06:52:37.134208 28422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 1 FOLLOWER]: Advancing to term 2
I20260621 06:52:37.135171 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.134824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:34696)
0621 06:52:37.134958 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.135050 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260621 06:52:37.135350 27873 leader_election.cc:334] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 85e295b70bac43c0a54c15c239835d0d (127.20.120.254:45135): Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
I20260621 06:52:37.136574 28422 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 06:52:37.139119 27707 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e64346b296436689045e4af52dd145 in term 2.
I20260621 06:52:37.140422 27871 leader_election.cc:302] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05e64346b296436689045e4af52dd145, 53b9942a875049a9a62d340b0257ee7b; no voters: 85e295b70bac43c0a54c15c239835d0d
I20260621 06:52:37.140694 28421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 06:52:37.141112 28421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [term 2 LEADER]: Becoming Leader. State: Replica: 05e64346b296436689045e4af52dd145, State: Running, Role: LEADER
I20260621 06:52:37.141477 28421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } }
I20260621 06:52:37.143229 28431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05e64346b296436689045e4af52dd145. Latest consensus state: current_term: 2 leader_uuid: "05e64346b296436689045e4af52dd145" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } } }
I20260621 06:52:37.143528 28431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:37.144094 28433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:37.147517 28433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:37.148311 28433 catalog_manager.cc:1259] Loaded cluster ID: 2660fb547c4648959bd33249505d04a5
I20260621 06:52:37.148415 28433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:37.149492 28433 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:37.150334 28433 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: Loaded TSK: 0
I20260621 06:52:37.151127 28433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:52:37.201743 27886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:52:37.208586 27886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:52:37.212613 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.212143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:34728)
0621 06:52:37.212352 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.212445 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 06:52:37.212579 27707 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [term 2 FOLLOWER]: Refusing update from remote peer 05e64346b296436689045e4af52dd145: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260621 06:52:37.213402 28431 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260621 06:52:37.213732 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.213329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:34730)
0621 06:52:37.213508 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.213570 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260621 06:52:37.213959 27873 consensus_peers.cc:487] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 -> Peer 85e295b70bac43c0a54c15c239835d0d (127.20.120.254:45135): Couldn't send request to peer 85e295b70bac43c0a54c15c239835d0d. Status: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 06:52:37.216097 28422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05e64346b296436689045e4af52dd145. Latest consensus state: current_term: 2 leader_uuid: "05e64346b296436689045e4af52dd145" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } } }
I20260621 06:52:37.216393 28422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:52:37.217731 28431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "05e64346b296436689045e4af52dd145" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } } }
I20260621 06:52:37.218046 28431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:37.218089 28435 mvcc.cc:205] Tried to move back new op lower bound from 7299173405536440320 to 7299173405255380992. Current Snapshot: MvccSnapshot[applied={T|T < 7299173405536440320}]
I20260621 06:52:37.223541 28422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "05e64346b296436689045e4af52dd145" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "85e295b70bac43c0a54c15c239835d0d" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45135 } } peers { permanent_uuid: "53b9942a875049a9a62d340b0257ee7b" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34695 } attrs { promote: false } } peers { permanent_uuid: "05e64346b296436689045e4af52dd145" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 34201 } attrs { promote: false } } }
I20260621 06:52:37.223850 28422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b9942a875049a9a62d340b0257ee7b [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:52:37.245355 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.227986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.20.120.193:40259)
0621 06:52:37.239790 (+ 11804us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:37.239799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:37.239825 (+ 26us) server_negotiation.cc:407] Connection header received
0621 06:52:37.239977 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:37.239984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:37.240132 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:37.240261 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:37.240806 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.241738 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.243699 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.244449 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.244613 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.244687 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:37.244929 (+ 242us) server_negotiation.cc:299] Negotiation successful
0621 06:52:37.245063 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11651,"thread_start_us":11552,"threads_started":1}
I20260621 06:52:37.245359 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.227905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:40259 (local address 127.0.0.1:38144)
0621 06:52:37.228102 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.228125 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:37.228267 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:37.240317 (+ 12050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:37.240325 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:37.240366 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:37.240620 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.240630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.241852 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.241860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.243534 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.243549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.244592 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.244599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.244781 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.244803 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 06:52:37.244902 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 06:52:37.245060 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260621 06:52:37.246347 27999 tablet_service.cc:1467] Processing CreateTablet for tablet 24cc7deb09bb4ad1b1cf3452aa777988 (DEFAULT_TABLE table=test-table [id=1fccebcfb59a48efb273831766c5946f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:37.246800 27999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cc7deb09bb4ad1b1cf3452aa777988. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:37.247627 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.228245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:43333 (local address 127.0.0.1:49582)
0621 06:52:37.228592 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.228609 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:37.228718 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:37.241246 (+ 12528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:37.241251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:37.241278 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:37.241548 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.241557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.243403 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.243411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.245240 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.245251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.247075 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.247083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.247210 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.247245 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 06:52:37.247315 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 06:52:37.247412 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260621 06:52:37.248624 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.228624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.20.120.194:43333)
0621 06:52:37.240892 (+ 12268us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:37.240900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:37.240922 (+ 22us) server_negotiation.cc:407] Connection header received
0621 06:52:37.241003 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:37.241010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:37.241096 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:37.241216 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:37.241713 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.243212 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.245951 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.246819 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.248105 (+ 1286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.248178 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:37.248227 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 06:52:37.248356 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12095,"thread_start_us":11970,"threads_started":1}
I20260621 06:52:37.249471 28143 tablet_service.cc:1467] Processing CreateTablet for tablet 24cc7deb09bb4ad1b1cf3452aa777988 (DEFAULT_TABLE table=test-table [id=1fccebcfb59a48efb273831766c5946f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:37.249945 28143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cc7deb09bb4ad1b1cf3452aa777988. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:37.251572 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.229040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.20.120.195:43025)
0621 06:52:37.244269 (+ 15229us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:37.244275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:37.244293 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:52:37.244373 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:37.244379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:37.244451 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:37.244552 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:37.245850 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.246869 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.249427 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.250599 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.250758 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.250831 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:37.251144 (+ 313us) server_negotiation.cc:299] Negotiation successful
0621 06:52:37.251284 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15097,"thread_start_us":15609,"threads_started":1}
I20260621 06:52:37.251595 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.228640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:43025 (local address 127.0.0.1:46622)
0621 06:52:37.238936 (+ 10296us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.238962 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:37.239113 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:37.245351 (+ 6238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:37.245354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:37.245372 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:37.245699 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.245709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.247026 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.247030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.249237 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.249250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.250793 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.250799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.250970 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.251004 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 06:52:37.251104 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 06:52:37.251240 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10119,"thread_start_us":10542,"threads_started":1}
I20260621 06:52:37.252945 28444 tablet_bootstrap.cc:492] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a: Bootstrap starting.
I20260621 06:52:37.254585 28286 tablet_service.cc:1467] Processing CreateTablet for tablet 24cc7deb09bb4ad1b1cf3452aa777988 (DEFAULT_TABLE table=test-table [id=1fccebcfb59a48efb273831766c5946f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:37.254752 28444 tablet_bootstrap.cc:654] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:37.254993 28286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cc7deb09bb4ad1b1cf3452aa777988. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:37.260205 28447 tablet_bootstrap.cc:492] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618: Bootstrap starting.
I20260621 06:52:37.261575 28447 tablet_bootstrap.cc:654] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:37.261694 28446 tablet_bootstrap.cc:492] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49: Bootstrap starting.
I20260621 06:52:37.263720 28447 tablet_bootstrap.cc:492] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618: No bootstrap required, opened a new log
I20260621 06:52:37.263916 28447 ts_tablet_manager.cc:1397] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260621 06:52:37.263923 28446 tablet_bootstrap.cc:654] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:37.264662 28444 tablet_bootstrap.cc:492] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a: No bootstrap required, opened a new log
I20260621 06:52:37.264886 28444 ts_tablet_manager.cc:1397] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.008s
I20260621 06:52:37.264798 28447 raft_consensus.cc:348] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:37.265142 28447 raft_consensus.cc:374] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:37.265290 28447 raft_consensus.cc:729] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c8fde0dfaf48099d75ed7dec484618, State: Initialized, Role: FOLLOWER
I20260621 06:52:37.265588 28447 consensus_queue.cc:260] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:37.266005 28444 raft_consensus.cc:348] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:37.266345 28444 raft_consensus.cc:374] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:37.266489 28444 raft_consensus.cc:729] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea92d7a226f4d5a8be376c02695cd4a, State: Initialized, Role: FOLLOWER
I20260621 06:52:37.266559 28447 ts_tablet_manager.cc:1428] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260621 06:52:37.266813 28444 consensus_queue.cc:260] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:37.269655 28351 heartbeater.cc:503] Master 127.20.120.252:34201 was elected leader, sending a full tablet report...
I20260621 06:52:37.266714 28446 tablet_bootstrap.cc:492] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49: No bootstrap required, opened a new log
I20260621 06:52:37.270258 28446 ts_tablet_manager.cc:1397] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260621 06:52:37.271097 28446 raft_consensus.cc:348] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:37.271384 28446 raft_consensus.cc:374] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:37.271505 28446 raft_consensus.cc:729] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8f4bce11874dd39985773f5e4a5e49, State: Initialized, Role: FOLLOWER
I20260621 06:52:37.271785 28446 consensus_queue.cc:260] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:37.272543 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.267173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.195:46047)
0621 06:52:37.272100 (+ 4927us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.272300 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4743,"thread_start_us":4637,"threads_started":1}
I20260621 06:52:37.272552 28446 ts_tablet_manager.cc:1428] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 06:52:37.274556 28208 heartbeater.cc:503] Master 127.20.120.252:34201 was elected leader, sending a full tablet report...
I20260621 06:52:37.274849 28444 ts_tablet_manager.cc:1428] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260621 06:52:37.277199 28064 heartbeater.cc:503] Master 127.20.120.252:34201 was elected leader, sending a full tablet report...
I20260621 06:52:37.282395 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.274394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.195:52867)
0621 06:52:37.282040 (+ 7646us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.282184 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":658}
I20260621 06:52:37.283699 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.275385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.193:50819)
0621 06:52:37.283369 (+ 7984us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.283502 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7868,"thread_start_us":7747,"threads_started":1}
I20260621 06:52:37.286666 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.272947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.194:38057)
0621 06:52:37.286345 (+ 13398us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.286480 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13271,"thread_start_us":13119,"threads_started":1}
I20260621 06:52:37.287571 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.287286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.194:33781)
0621 06:52:37.287418 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.287477 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 06:52:37.336345 28453 raft_consensus.cc:482] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:52:37.336562 28453 raft_consensus.cc:504] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:37.338275 28453 leader_election.cc:288] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dea92d7a226f4d5a8be376c02695cd4a (127.20.120.193:40259), f7c8fde0dfaf48099d75ed7dec484618 (127.20.120.195:43025)
I20260621 06:52:37.344456 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.338693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:47663 (local address 127.20.120.193:40259)
0621 06:52:37.338890 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:37.338896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:37.338925 (+ 29us) server_negotiation.cc:407] Connection header received
0621 06:52:37.339153 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:37.339160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:37.339244 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:37.339358 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:37.339888 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.340752 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.342632 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.343444 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.343614 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.343721 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:37.344063 (+ 342us) server_negotiation.cc:299] Negotiation successful
0621 06:52:37.344184 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 06:52:37.344491 28454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.338576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:40259 (local address 127.20.120.194:47663)
0621 06:52:37.338755 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.338778 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:37.338963 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:37.339383 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:37.339386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:37.339406 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:37.339745 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.339757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.340894 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.340899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.342476 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.342488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.343663 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.343669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.343882 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.343928 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 06:52:37.344038 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 06:52:37.344175 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":44}
I20260621 06:52:37.344527 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.338620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:35825 (local address 127.20.120.195:43025)
0621 06:52:37.338843 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:37.338850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:37.339498 (+ 648us) server_negotiation.cc:407] Connection header received
0621 06:52:37.339659 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:37.339667 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:37.339742 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:37.339841 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:37.340384 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.341133 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.342815 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.343655 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.343816 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.343886 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:37.344149 (+ 263us) server_negotiation.cc:299] Negotiation successful
0621 06:52:37.344273 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 06:52:37.344656 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.338576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:43025 (local address 127.20.120.194:35825)
0621 06:52:37.339394 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.339412 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:37.339529 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:37.339869 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:37.339873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:37.339895 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:37.340236 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.340247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.341256 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.341260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.342679 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:37.342692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.343870 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.343875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:37.344002 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.344024 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 06:52:37.344124 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 06:52:37.344239 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":222,"thread_start_us":324,"threads_started":1}
I20260621 06:52:37.345295 28305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc7deb09bb4ad1b1cf3452aa777988" candidate_uuid: "0b8f4bce11874dd39985773f5e4a5e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c8fde0dfaf48099d75ed7dec484618" is_pre_election: true
I20260621 06:52:37.345590 28305 raft_consensus.cc:2454] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b8f4bce11874dd39985773f5e4a5e49 in term 0.
I20260621 06:52:37.345629 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc7deb09bb4ad1b1cf3452aa777988" candidate_uuid: "0b8f4bce11874dd39985773f5e4a5e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea92d7a226f4d5a8be376c02695cd4a" is_pre_election: true
I20260621 06:52:37.346048 28019 raft_consensus.cc:2454] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b8f4bce11874dd39985773f5e4a5e49 in term 0.
I20260621 06:52:37.346097 28099 leader_election.cc:302] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8f4bce11874dd39985773f5e4a5e49, f7c8fde0dfaf48099d75ed7dec484618; no voters:
I20260621 06:52:37.346486 28453 raft_consensus.cc:2781] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:52:37.346601 28453 raft_consensus.cc:482] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:52:37.346700 28453 raft_consensus.cc:3037] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:37.348749 28453 raft_consensus.cc:504] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:37.349339 28453 leader_election.cc:288] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [CANDIDATE]: Term 1 election: Requested vote from peers dea92d7a226f4d5a8be376c02695cd4a (127.20.120.193:40259), f7c8fde0dfaf48099d75ed7dec484618 (127.20.120.195:43025)
I20260621 06:52:37.349810 28019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc7deb09bb4ad1b1cf3452aa777988" candidate_uuid: "0b8f4bce11874dd39985773f5e4a5e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea92d7a226f4d5a8be376c02695cd4a"
I20260621 06:52:37.349830 28305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc7deb09bb4ad1b1cf3452aa777988" candidate_uuid: "0b8f4bce11874dd39985773f5e4a5e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c8fde0dfaf48099d75ed7dec484618"
I20260621 06:52:37.350044 28305 raft_consensus.cc:3037] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:37.350044 28019 raft_consensus.cc:3037] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:37.353017 28305 raft_consensus.cc:2454] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b8f4bce11874dd39985773f5e4a5e49 in term 1.
I20260621 06:52:37.353018 28019 raft_consensus.cc:2454] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b8f4bce11874dd39985773f5e4a5e49 in term 1.
I20260621 06:52:37.353468 28096 leader_election.cc:302] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b8f4bce11874dd39985773f5e4a5e49, dea92d7a226f4d5a8be376c02695cd4a; no voters:
I20260621 06:52:37.353762 28453 raft_consensus.cc:2781] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:37.354254 28453 raft_consensus.cc:686] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 1 LEADER]: Becoming Leader. State: Replica: 0b8f4bce11874dd39985773f5e4a5e49, State: Running, Role: LEADER
I20260621 06:52:37.354667 28453 consensus_queue.cc:237] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } }
I20260621 06:52:37.355350 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.354864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.194:48177)
0621 06:52:37.355092 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.355184 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260621 06:52:37.358207 27885 catalog_manager.cc:5495] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b8f4bce11874dd39985773f5e4a5e49 (127.20.120.194). New cstate: current_term: 1 leader_uuid: "0b8f4bce11874dd39985773f5e4a5e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b8f4bce11874dd39985773f5e4a5e49" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 } health_report { overall_health: UNKNOWN } } }
I20260621 06:52:37.390030 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.384852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.20.120.194:43333)
0621 06:52:37.385074 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:37.385080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:37.385099 (+ 19us) server_negotiation.cc:407] Connection header received
0621 06:52:37.385289 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:37.385294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:37.385377 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:37.385540 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:52:37.385873 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.386607 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.387893 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.388320 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.388458 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.388847 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:52:37.389374 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:52:37.389533 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:37.389609 (+ 76us) server_negotiation.cc:299] Negotiation successful
0621 06:52:37.389691 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 06:52:37.394179 28305 raft_consensus.cc:1261] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618 [term 1 FOLLOWER]: Refusing update from remote peer 0b8f4bce11874dd39985773f5e4a5e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:37.394613 28019 raft_consensus.cc:1261] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a [term 1 FOLLOWER]: Refusing update from remote peer 0b8f4bce11874dd39985773f5e4a5e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:37.394928 28457 consensus_queue.cc:1035] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c8fde0dfaf48099d75ed7dec484618" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:37.395323 28453 consensus_queue.cc:1035] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dea92d7a226f4d5a8be376c02695cd4a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 40259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:37.398139 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.397137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.195:53609)
0621 06:52:37.397777 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.397943 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491}
I20260621 06:52:37.398679 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.397450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.193:44047)
0621 06:52:37.398425 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.398517 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":823}
I20260621 06:52:37.403888 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.403429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.195:49743)
0621 06:52:37.403652 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.403729 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260621 06:52:37.408939 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.408396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.193:56129)
0621 06:52:37.408561 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.408692 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 06:52:37.410048 28463 mvcc.cc:205] Tried to move back new op lower bound from 7299173406279045120 to 7299173406128746496. Current Snapshot: MvccSnapshot[applied={T|T < 7299173406279045120}]
I20260621 06:52:37.418053 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.417583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.194:50009)
0621 06:52:37.417738 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.417813 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 06:52:37.419446 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.419055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.20.120.194:44649)
0621 06:52:37.419182 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.419269 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260621 06:52:37.426164 27885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:37.426458 27885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:37.431795 27885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05e64346b296436689045e4af52dd145: Sending DeleteTablet for 3 replicas of tablet 24cc7deb09bb4ad1b1cf3452aa777988
I20260621 06:52:37.432837 27999 tablet_service.cc:1514] Processing DeleteTablet for tablet 24cc7deb09bb4ad1b1cf3452aa777988 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:37 UTC) from {username='oryx', principal='oryx/127.20.120.252@KRBTEST.COM'} at 127.0.0.1:38144
I20260621 06:52:37.432971 28286 tablet_service.cc:1514] Processing DeleteTablet for tablet 24cc7deb09bb4ad1b1cf3452aa777988 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:37 UTC) from {username='oryx', principal='oryx/127.20.120.252@KRBTEST.COM'} at 127.0.0.1:46622
I20260621 06:52:37.432996 28143 tablet_service.cc:1514] Processing DeleteTablet for tablet 24cc7deb09bb4ad1b1cf3452aa777988 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:37 UTC) from {username='oryx', principal='oryx/127.20.120.252@KRBTEST.COM'} at 127.0.0.1:49582
I20260621 06:52:37.445242 28470 tablet_replica.cc:330] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a: stopping tablet replica
I20260621 06:52:37.445498 28470 raft_consensus.cc:2229] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:37.445855 28470 raft_consensus.cc:2258] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:37.446514 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 27858
I20260621 06:52:37.447127 28470 ts_tablet_manager.cc:1905] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:37.455230 28470 ts_tablet_manager.cc:1918] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:37.459092 28471 tablet_replica.cc:330] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49: stopping tablet replica
I20260621 06:52:37.459437 28471 raft_consensus.cc:2229] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:52:37.460177 28471 raft_consensus.cc:2258] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:37.460834 28472 tablet_replica.cc:330] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618: stopping tablet replica
I20260621 06:52:37.461114 28472 raft_consensus.cc:2229] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:37.461792 28470 log.cc:1196] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/24cc7deb09bb4ad1b1cf3452aa777988
I20260621 06:52:37.461825 28471 ts_tablet_manager.cc:1905] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:37.462476 28470 ts_tablet_manager.cc:1939] T 24cc7deb09bb4ad1b1cf3452aa777988 P dea92d7a226f4d5a8be376c02695cd4a: Deleting consensus metadata
I20260621 06:52:37.463557 27870 catalog_manager.cc:4841] TS dea92d7a226f4d5a8be376c02695cd4a (127.20.120.193:40259): tablet 24cc7deb09bb4ad1b1cf3452aa777988 (table test-table [id=1fccebcfb59a48efb273831766c5946f]) successfully deleted
I20260621 06:52:37.464087 28472 raft_consensus.cc:2258] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:37.466761 28472 ts_tablet_manager.cc:1905] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:37.469431 28471 ts_tablet_manager.cc:1918] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:37.469617 28471 log.cc:1196] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/24cc7deb09bb4ad1b1cf3452aa777988
I20260621 06:52:37.470173 28471 ts_tablet_manager.cc:1939] T 24cc7deb09bb4ad1b1cf3452aa777988 P 0b8f4bce11874dd39985773f5e4a5e49: Deleting consensus metadata
I20260621 06:52:37.471230 27870 catalog_manager.cc:4841] TS 0b8f4bce11874dd39985773f5e4a5e49 (127.20.120.194:43333): tablet 24cc7deb09bb4ad1b1cf3452aa777988 (table test-table [id=1fccebcfb59a48efb273831766c5946f]) successfully deleted
I20260621 06:52:37.472362 28472 ts_tablet_manager.cc:1918] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:37.472498 28472 log.cc:1196] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/24cc7deb09bb4ad1b1cf3452aa777988
I20260621 06:52:37.472929 28472 ts_tablet_manager.cc:1939] T 24cc7deb09bb4ad1b1cf3452aa777988 P f7c8fde0dfaf48099d75ed7dec484618: Deleting consensus metadata
I20260621 06:52:37.473430 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.447769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38156 (local address 127.20.120.193:40259)
0621 06:52:37.447980 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:37.447989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:37.455977 (+ 7988us) server_negotiation.cc:407] Connection header received
0621 06:52:37.456286 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:37.456294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:37.456396 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:37.456476 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:37.458900 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.459915 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.461714 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.462382 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.462529 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.466056 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:37.466074 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:37.466093 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:37.466128 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:37.470085 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:37.470886 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:37.470895 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:37.470899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:37.471041 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:37.471503 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:37.471508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:37.471511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:37.471925 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:37.472239 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:37.472819 (+ 580us) server_negotiation.cc:299] Negotiation successful
0621 06:52:37.473048 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 06:52:37.474056 28059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:37.474417 27873 catalog_manager.cc:4841] TS f7c8fde0dfaf48099d75ed7dec484618 (127.20.120.195:43025): tablet 24cc7deb09bb4ad1b1cf3452aa777988 (table test-table [id=1fccebcfb59a48efb273831766c5946f]) successfully deleted
I20260621 06:52:37.603796 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 27858
I20260621 06:52:37.633379 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 28075
I20260621 06:52:37.648155 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.634171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49610 (local address 127.20.120.194:43333)
0621 06:52:37.634355 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:37.634366 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:37.634390 (+ 24us) server_negotiation.cc:407] Connection header received
0621 06:52:37.634554 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:37.634561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:37.634668 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:37.634785 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:37.636164 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.637441 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.638797 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.639417 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.639565 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.641876 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:37.641889 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:37.641929 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:37.641970 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:37.645534 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:37.646284 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:37.646293 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:37.646296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:37.646463 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:37.646759 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:37.646763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:37.646767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:37.647156 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:37.647430 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:37.647587 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 06:52:37.647776 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 06:52:37.648813 28203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:37.764006 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 28075
I20260621 06:52:37.782728 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 28221
I20260621 06:52:37.795111 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.783432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.20.120.195:43025)
0621 06:52:37.783612 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:37.783620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:37.783665 (+ 45us) server_negotiation.cc:407] Connection header received
0621 06:52:37.783832 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:37.783835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:37.783908 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:37.784002 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:37.785417 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.786212 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.787416 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:37.787796 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:37.787884 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:37.790365 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:37.790381 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:37.790405 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:37.790432 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:37.792920 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:37.793475 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:37.793481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:37.793483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:37.793584 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:37.793882 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:37.793885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:37.793886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:37.794254 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:37.794482 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:37.794711 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 06:52:37.794860 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 06:52:37.795660 28346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:37.809378 28477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.808738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:34734)
0621 06:52:37.809152 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.809229 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":213,"threads_started":1}
I20260621 06:52:37.810913 28477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:37.810554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45135 (local address 127.0.0.1:34744)
0621 06:52:37.810689 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:37.810763 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:45135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 06:52:37.887084 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 28221
I20260621 06:52:37.903903 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 27561
I20260621 06:52:37.904904 27727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:38.018939 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 27561
I20260621 06:52:38.039227 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 27661
I20260621 06:52:38.040261 27916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:38.159123 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 27661
2026-06-21T06:52:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9583 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:52:38 dist-test-slave-gmf2 krb5kdc[28483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:52:38 dist-test-slave-gmf2 krb5kdc[28483](info): set up 2 sockets
Jun 21 06:52:38 dist-test-slave-gmf2 krb5kdc[28483](info): commencing operation
krb5kdc: starting...
W20260621 06:52:39.243152 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:52:39 dist-test-slave-gmf2 krb5kdc[28483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.254.keytab.
Entry for principal oryx/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.254.keytab.
I20260621 06:52:39.454270 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:39173
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.254.keytab
--principal=oryx/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:39173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:39.768329 28496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:39.768637 28496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:39.768754 28496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:39.776093 28496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:39.776208 28496 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 06:52:39.776258 28496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:39.776351 28496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:39.776439 28496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:39.776541 28496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:39.786132 28496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:39173
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.254.keytab
--principal=oryx/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:39.787451 28496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:39.788791 28496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:39.801312 28501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:39.801518 28504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:39.801539 28502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:39.802115 28496 server_base.cc:1034] running on GCE node
I20260621 06:52:39.802996 28496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:39.804401 28496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:39.805616 28496 hybrid_clock.cc:648] HybridClock initialized: now 1782024759805556 us; error 35 us; skew 500 ppm
I20260621 06:52:39.805998 28496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:39 dist-test-slave-gmf2 krb5kdc[28483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024759, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:39.808979 28496 init.cc:377] Logged in from keytab as oryx/127.20.120.254@KRBTEST.COM (short username oryx)
I20260621 06:52:39.810449 28496 webserver.cc:460] Webserver started at http://127.20.120.254:45387/ using document root <none> and password file <none>
I20260621 06:52:39.811103 28496 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:39.811188 28496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:39.811462 28496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:39.814409 28496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "569e664927304215931d87758f4910eb"
format_stamp: "Formatted at 2026-06-21 06:52:39 on dist-test-slave-gmf2"
server_key: "2b521cb1adbae758737242e38335d9ad"
server_key_iv: "1bbb6945a10262881646f076817395ae"
server_key_version: "encryptionkey@0"
I20260621 06:52:39.815135 28496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "569e664927304215931d87758f4910eb"
format_stamp: "Formatted at 2026-06-21 06:52:39 on dist-test-slave-gmf2"
server_key: "2b521cb1adbae758737242e38335d9ad"
server_key_iv: "1bbb6945a10262881646f076817395ae"
server_key_version: "encryptionkey@0"
I20260621 06:52:39.819972 28496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:52:39.823297 28511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:39.824651 28496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:52:39.824868 28496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "569e664927304215931d87758f4910eb"
format_stamp: "Formatted at 2026-06-21 06:52:39 on dist-test-slave-gmf2"
server_key: "2b521cb1adbae758737242e38335d9ad"
server_key_iv: "1bbb6945a10262881646f076817395ae"
server_key_version: "encryptionkey@0"
I20260621 06:52:39.825104 28496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:39.840583 28496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:39.844602 28496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:39.844906 28496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:39.866573 28496 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:39173
I20260621 06:52:39.866597 28572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:39173 every 8 connection(s)
I20260621 06:52:39.867956 28496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:39.871919 28573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:39.873101 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 28496
I20260621 06:52:39.873310 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:39.873986 20963 external_mini_cluster.cc:1428] Setting key 0178369b8790cd72595868c9a91ff387
I20260621 06:52:39.885141 28573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb: Bootstrap starting.
I20260621 06:52:39.889142 28573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb: Neither blocks nor log segments found. Creating new log.
Jun 21 06:52:39 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:39.890499 28573 log.cc:824] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:39.893999 28573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb: No bootstrap required, opened a new log
I20260621 06:52:39.899070 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:39.876061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.20.120.254:39173)
0621 06:52:39.877094 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:39.877112 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:39.877524 (+ 412us) server_negotiation.cc:407] Connection header received
0621 06:52:39.880091 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:39.880119 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:39.880616 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:39.881091 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:39.882280 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:39.884272 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:39.885664 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:39.886773 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:39.887026 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:39.891180 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:39.891188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:39.891202 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:39.891235 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:39.894613 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:39.895334 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:39.895343 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:39.895349 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:39.895531 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:39.895958 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:39.895964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:39.895967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:39.896567 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:39.896893 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:39.897459 (+ 566us) server_negotiation.cc:299] Negotiation successful
0621 06:52:39.897875 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":450,"threads_started":1}
I20260621 06:52:39.902402 28573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e664927304215931d87758f4910eb" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39173 } }
I20260621 06:52:39.902827 28573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:39.902940 28573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569e664927304215931d87758f4910eb, State: Initialized, Role: FOLLOWER
I20260621 06:52:39.903530 28573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e664927304215931d87758f4910eb" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39173 } }
I20260621 06:52:39.903800 28573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:39.903928 28573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:39.904090 28573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:39.906672 28573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e664927304215931d87758f4910eb" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39173 } }
I20260621 06:52:39.907181 28573 leader_election.cc:302] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 569e664927304215931d87758f4910eb; no voters:
I20260621 06:52:39.907749 28573 leader_election.cc:288] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:39.907922 28578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:39.908615 28578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [term 1 LEADER]: Becoming Leader. State: Replica: 569e664927304215931d87758f4910eb, State: Running, Role: LEADER
I20260621 06:52:39.909157 28578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e664927304215931d87758f4910eb" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39173 } }
I20260621 06:52:39.909878 28573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:39.911809 28580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 569e664927304215931d87758f4910eb. Latest consensus state: current_term: 1 leader_uuid: "569e664927304215931d87758f4910eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e664927304215931d87758f4910eb" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39173 } } }
I20260621 06:52:39.912457 28580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:39.912206 28579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "569e664927304215931d87758f4910eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e664927304215931d87758f4910eb" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39173 } } }
I20260621 06:52:39.912679 28579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:39.913429 28585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:39.920526 28585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:39.929689 28585 catalog_manager.cc:1347] Generated new cluster ID: 3cc7e6bd81564762981667a8e0110a10
I20260621 06:52:39.929809 28585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:39.938926 28585 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:39.940783 28585 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:39.948633 28585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb: Generated new TSK 0
I20260621 06:52:39.949600 28585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab.
Entry for principal oryx/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab.
I20260621 06:52:40.009651 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:39173
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab
--principal=oryx/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:40.337688 28599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:40.338018 28599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:40.338115 28599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:40.344930 28599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:40.345024 28599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:40.345103 28599 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 06:52:40.345170 28599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:52:40.354600 28599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab
--principal=oryx/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.120.254:39173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:40.355790 28599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:40.357000 28599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:40.369829 28604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:40.369801 28605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:40.369855 28607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:40.370662 28599 server_base.cc:1034] running on GCE node
I20260621 06:52:40.371312 28599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:40.372225 28599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:40.373467 28599 hybrid_clock.cc:648] HybridClock initialized: now 1782024760373391 us; error 45 us; skew 500 ppm
I20260621 06:52:40.373767 28599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:40 dist-test-slave-gmf2 krb5kdc[28483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024760, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:40.376860 28599 init.cc:377] Logged in from keytab as oryx/127.20.120.193@KRBTEST.COM (short username oryx)
I20260621 06:52:40.378273 28599 webserver.cc:460] Webserver started at http://127.20.120.193:35715/ using document root <none> and password file <none>
I20260621 06:52:40.378899 28599 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:40.378978 28599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:40.379263 28599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:40.382071 28599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "9da34b0cd81c46138c244cffa93469d3"
format_stamp: "Formatted at 2026-06-21 06:52:40 on dist-test-slave-gmf2"
server_key: "4f69eca90ebdc9550013ab5bca8baf0a"
server_key_iv: "e451c2fac2187d4bbf2348c44a8a363c"
server_key_version: "encryptionkey@0"
I20260621 06:52:40.382823 28599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "9da34b0cd81c46138c244cffa93469d3"
format_stamp: "Formatted at 2026-06-21 06:52:40 on dist-test-slave-gmf2"
server_key: "4f69eca90ebdc9550013ab5bca8baf0a"
server_key_iv: "e451c2fac2187d4bbf2348c44a8a363c"
server_key_version: "encryptionkey@0"
I20260621 06:52:40.387722 28599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:52:40.391098 28614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:40.392377 28599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:52:40.392568 28599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "9da34b0cd81c46138c244cffa93469d3"
format_stamp: "Formatted at 2026-06-21 06:52:40 on dist-test-slave-gmf2"
server_key: "4f69eca90ebdc9550013ab5bca8baf0a"
server_key_iv: "e451c2fac2187d4bbf2348c44a8a363c"
server_key_version: "encryptionkey@0"
I20260621 06:52:40.392784 28599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:40.414670 28599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:40.418768 28599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:40.419024 28599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:40.421602 28599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:40.421744 28599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260621 06:52:40.421927 28599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:40.422034 28599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 06:52:40 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024760, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:40.448458 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:40.423148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:48691 (local address 127.20.120.254:39173)
0621 06:52:40.423626 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:40.423633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:40.424626 (+ 993us) server_negotiation.cc:407] Connection header received
0621 06:52:40.427485 (+ 2859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:40.427494 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:40.427600 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:40.427695 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:40.430522 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:40.431629 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:40.434932 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:40.435379 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:40.435541 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:40.440286 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:40.440296 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:40.440304 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:40.440360 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:40.444323 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:40.445147 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:40.445154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:40.445158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:40.445319 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:40.445832 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:40.445839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:40.445843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:40.446214 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:40.446425 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:40.447559 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0621 06:52:40.447857 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":360,"threads_started":1}
I20260621 06:52:40.450397 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:40.423485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:39173 (local address 127.20.120.193:48691)
0621 06:52:40.424448 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:40.424499 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:40.427105 (+ 2606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:40.428319 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:40.428362 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:40.429306 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:40.430315 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:40.430345 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:40.431854 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:40.431863 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:40.434663 (+ 2800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:40.434676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:40.435643 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:40.435654 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:40.435996 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:40.436764 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:40.436801 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:40.440006 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:40.444533 (+ 4527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:40.444543 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:40.444561 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:40.445014 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:40.445448 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:40.445454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:40.445459 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:40.445695 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:40.446396 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:40.446407 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:40.446982 (+ 575us) client_negotiation.cc:769] Sending connection context
0621 06:52:40.447415 (+ 433us) client_negotiation.cc:240] Negotiation successful
0621 06:52:40.448128 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":379,"threads_started":1}
I20260621 06:52:40.472113 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:40.458947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:37531 (local address 127.20.120.254:39173)
0621 06:52:40.459117 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:40.459122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:40.459367 (+ 245us) server_negotiation.cc:407] Connection header received
0621 06:52:40.459587 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:40.459595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:40.459687 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:40.459797 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:40.461360 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:40.462236 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:40.463672 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:40.464074 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:40.464197 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:40.466078 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:40.466083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:40.466086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:40.466113 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:40.469149 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:40.469864 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:40.469872 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:40.469876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:40.470036 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:40.470458 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:40.470467 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:40.470471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:40.470884 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:40.471073 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:40.471516 (+ 443us) server_negotiation.cc:299] Negotiation successful
0621 06:52:40.471736 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 06:52:40.472239 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:40.458454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:39173 (local address 127.20.120.193:37531)
0621 06:52:40.459243 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:40.459269 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:40.459441 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:40.459949 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:40.459953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:40.460638 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:40.461159 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:40.461172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:40.462370 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:40.462375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:40.463542 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:40.463553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:40.464256 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:40.464262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:40.464427 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:40.465146 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:40.465162 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:40.465856 (+ 694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:40.469272 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:40.469278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:40.469285 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:40.469753 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:40.470139 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:40.470148 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:40.470152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:40.470313 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:40.471048 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:40.471055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:40.471265 (+ 210us) client_negotiation.cc:769] Sending connection context
0621 06:52:40.471433 (+ 168us) client_negotiation.cc:240] Negotiation successful
0621 06:52:40.471761 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":378,"threads_started":1}
I20260621 06:52:40.476524 28599 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:38051
I20260621 06:52:40.476524 28734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:38051 every 8 connection(s)
I20260621 06:52:40.477810 28599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:52:40.479720 28735 heartbeater.cc:347] Connected to a master server at 127.20.120.254:39173
I20260621 06:52:40.479951 28735 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:40.480557 28735 heartbeater.cc:511] Master 127.20.120.254:39173 requested a full tablet report, sending...
I20260621 06:52:40.482476 28528 ts_manager.cc:194] Registered new tserver with Master: 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193:38051)
I20260621 06:52:40.483770 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 28599
I20260621 06:52:40.483839 28528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:48691
I20260621 06:52:40.483978 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:52:40.484606 20963 external_mini_cluster.cc:1428] Setting key 6543c6832497e37f2a398171e0a18520
WARNING: no policy specified for oryx/127.20.120.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab.
Entry for principal oryx/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab.
I20260621 06:52:40.528268 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:39173
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab
--principal=oryx/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:40.873737 28739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:40.874125 28739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:40.874281 28739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:40.885659 28739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:40.885844 28739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:40.886057 28739 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 06:52:40.886209 28739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:52:40.897575 28739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab
--principal=oryx/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.120.254:39173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:40.898859 28739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:40.900151 28739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:40.913275 28747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:40.913316 28744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:40.913288 28745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:40.914486 28739 server_base.cc:1034] running on GCE node
I20260621 06:52:40.915079 28739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:40.916031 28739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:40.917279 28739 hybrid_clock.cc:648] HybridClock initialized: now 1782024760917211 us; error 46 us; skew 500 ppm
I20260621 06:52:40.917593 28739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:40 dist-test-slave-gmf2 krb5kdc[28483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024760, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:40.920652 28739 init.cc:377] Logged in from keytab as oryx/127.20.120.194@KRBTEST.COM (short username oryx)
I20260621 06:52:40.922133 28739 webserver.cc:460] Webserver started at http://127.20.120.194:32901/ using document root <none> and password file <none>
I20260621 06:52:40.922931 28739 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:40.923065 28739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:40.923380 28739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:40.926433 28739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "f7f284c245134cc3b89c862385b1e222"
format_stamp: "Formatted at 2026-06-21 06:52:40 on dist-test-slave-gmf2"
server_key: "d70817a7cbc0a0c2c3efaec4f1cdc686"
server_key_iv: "5df37542c82fe04ee9bf27aa4c6a0f3f"
server_key_version: "encryptionkey@0"
I20260621 06:52:40.927260 28739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "f7f284c245134cc3b89c862385b1e222"
format_stamp: "Formatted at 2026-06-21 06:52:40 on dist-test-slave-gmf2"
server_key: "d70817a7cbc0a0c2c3efaec4f1cdc686"
server_key_iv: "5df37542c82fe04ee9bf27aa4c6a0f3f"
server_key_version: "encryptionkey@0"
I20260621 06:52:40.932668 28739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260621 06:52:40.936228 28754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:40.937690 28739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 06:52:40.937846 28739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "f7f284c245134cc3b89c862385b1e222"
format_stamp: "Formatted at 2026-06-21 06:52:40 on dist-test-slave-gmf2"
server_key: "d70817a7cbc0a0c2c3efaec4f1cdc686"
server_key_iv: "5df37542c82fe04ee9bf27aa4c6a0f3f"
server_key_version: "encryptionkey@0"
I20260621 06:52:40.938113 28739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:40.952473 28739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:40.956478 28739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:40.956748 28739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:40.959311 28739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:40.959420 28739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:40.959576 28739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:40.959653 28739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 06:52:40 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024760, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:40.984285 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:40.960051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:45733 (local address 127.20.120.254:39173)
0621 06:52:40.960247 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:40.960253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:40.961763 (+ 1510us) server_negotiation.cc:407] Connection header received
0621 06:52:40.964400 (+ 2637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:40.964405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:40.964514 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:40.964626 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:40.966937 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:40.968264 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:40.970660 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:40.971289 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:40.971431 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:40.977067 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:40.977077 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:40.977083 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:40.977140 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:40.980872 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:40.981634 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:40.981639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:40.981643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:40.981768 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:40.982318 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:40.982323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:40.982326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:40.982606 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:40.982755 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:40.983720 (+ 965us) server_negotiation.cc:299] Negotiation successful
0621 06:52:40.983950 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 06:52:40.986004 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:40.960677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:39173 (local address 127.20.120.194:45733)
0621 06:52:40.961575 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:40.961628 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:40.963970 (+ 2342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:40.964858 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:40.964887 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:40.965722 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:40.966709 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:40.966742 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:40.968475 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:40.968487 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:40.970460 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:40.970476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:40.971550 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:40.971559 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:40.971819 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:40.972843 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:40.972882 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:40.976696 (+ 3814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:40.981036 (+ 4340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:40.981046 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:40.981064 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:40.981496 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:40.981880 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:40.981886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:40.981890 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:40.982181 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:40.982750 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:40.982763 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:40.983265 (+ 502us) client_negotiation.cc:769] Sending connection context
0621 06:52:40.983622 (+ 357us) client_negotiation.cc:240] Negotiation successful
0621 06:52:40.984239 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":420,"threads_started":1}
I20260621 06:52:41.005752 28739 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:44587
I20260621 06:52:41.005776 28873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:44587 every 8 connection(s)
I20260621 06:52:41.007184 28739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:52:41.008368 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 28739
I20260621 06:52:41.008562 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:52:41.009260 20963 external_mini_cluster.cc:1428] Setting key fd223d8de1ea8ae8e9c584eedbe7ecac
I20260621 06:52:41.009684 28874 heartbeater.cc:347] Connected to a master server at 127.20.120.254:39173
I20260621 06:52:41.009943 28874 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:41.012060 28874 heartbeater.cc:511] Master 127.20.120.254:39173 requested a full tablet report, sending...
I20260621 06:52:41.012962 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:40.993506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:36745 (local address 127.20.120.254:39173)
0621 06:52:40.993716 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:40.993724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:40.998703 (+ 4979us) server_negotiation.cc:407] Connection header received
0621 06:52:40.998915 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:40.998919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:40.998989 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:40.999112 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:41.000059 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.001061 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.002716 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.003515 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.003651 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.006408 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:41.006413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:41.006417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:41.006448 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:41.009826 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:41.010750 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:41.010754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:41.010757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:41.010909 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:41.011357 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:41.011361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:41.011363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:41.011770 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:41.011921 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.012412 (+ 491us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.012631 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:52:41.013221 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:40.993663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:39173 (local address 127.20.120.194:36745)
0621 06:52:40.998547 (+ 4884us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:40.998575 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:40.998784 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:40.999150 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:40.999154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:40.999613 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:40.999895 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:40.999903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.001194 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.001201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.002520 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.002558 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.003883 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.003890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.004058 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.004805 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:41.004832 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:41.006233 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:41.010035 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:41.010041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:41.010046 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:41.010590 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:41.011049 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:41.011053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:41.011056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:41.011215 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:41.011907 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:41.011912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:41.012143 (+ 231us) client_negotiation.cc:769] Sending connection context
0621 06:52:41.012293 (+ 150us) client_negotiation.cc:240] Negotiation successful
0621 06:52:41.012637 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4624,"thread_start_us":4411,"threads_started":1}
I20260621 06:52:41.013567 28528 ts_manager.cc:194] Registered new tserver with Master: f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587)
I20260621 06:52:41.014930 28528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:45733
WARNING: no policy specified for oryx/127.20.120.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab.
Entry for principal oryx/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab.
I20260621 06:52:41.057073 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:39173
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab
--principal=oryx/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:41.388661 28878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:41.388995 28878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:41.389192 28878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:41.396461 28878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:41.396610 28878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:41.396759 28878 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 06:52:41.396896 28878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:52:41.407132 28878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab
--principal=oryx/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.120.254:39173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:41.408499 28878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:41.409897 28878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:41.423488 28883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:41.423405 28884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:41.423375 28886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:41.424203 28878 server_base.cc:1034] running on GCE node
I20260621 06:52:41.424855 28878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:41.425733 28878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:41.426942 28878 hybrid_clock.cc:648] HybridClock initialized: now 1782024761426876 us; error 74 us; skew 500 ppm
I20260621 06:52:41.427206 28878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:41 dist-test-slave-gmf2 krb5kdc[28483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024761, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:41.430460 28878 init.cc:377] Logged in from keytab as oryx/127.20.120.195@KRBTEST.COM (short username oryx)
I20260621 06:52:41.431898 28878 webserver.cc:460] Webserver started at http://127.20.120.195:37307/ using document root <none> and password file <none>
I20260621 06:52:41.432613 28878 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:41.432729 28878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:41.433068 28878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:41.436898 28878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "e3ad1765a77e44678de168473f3ccffe"
format_stamp: "Formatted at 2026-06-21 06:52:41 on dist-test-slave-gmf2"
server_key: "e612f145e2262c34c8a7fbeb0dca8b21"
server_key_iv: "9ef7e58c8ae86a16ca8106daa67d55bb"
server_key_version: "encryptionkey@0"
I20260621 06:52:41.438213 28878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "e3ad1765a77e44678de168473f3ccffe"
format_stamp: "Formatted at 2026-06-21 06:52:41 on dist-test-slave-gmf2"
server_key: "e612f145e2262c34c8a7fbeb0dca8b21"
server_key_iv: "9ef7e58c8ae86a16ca8106daa67d55bb"
server_key_version: "encryptionkey@0"
I20260621 06:52:41.445243 28878 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260621 06:52:41.449286 28893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:41.450783 28878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:41.450996 28878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "e3ad1765a77e44678de168473f3ccffe"
format_stamp: "Formatted at 2026-06-21 06:52:41 on dist-test-slave-gmf2"
server_key: "e612f145e2262c34c8a7fbeb0dca8b21"
server_key_iv: "9ef7e58c8ae86a16ca8106daa67d55bb"
server_key_version: "encryptionkey@0"
I20260621 06:52:41.451241 28878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:41.474213 28878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:41.478720 28878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:41.479003 28878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:41.481717 28878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:41.482113 28878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260621 06:52:41.482290 28878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:41.482447 28878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:41.486979 28735 heartbeater.cc:503] Master 127.20.120.254:39173 was elected leader, sending a full tablet report...
Jun 21 06:52:41 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024761, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:41.507526 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.482635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:42677 (local address 127.20.120.254:39173)
0621 06:52:41.482832 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.482838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.484321 (+ 1483us) server_negotiation.cc:407] Connection header received
0621 06:52:41.486530 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.486536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.486627 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.486752 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:41.489348 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.490446 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.493248 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.493959 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.494124 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.499391 (+ 5267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:41.499400 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:41.499407 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:41.499455 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:41.503236 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:41.504388 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:41.504394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:41.504398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:41.504571 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:41.505277 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:41.505281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:41.505283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:41.505674 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:41.505809 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.506915 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.507178 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 06:52:41.508881 28902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.483241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:39173 (local address 127.20.120.195:42677)
0621 06:52:41.484125 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:41.484179 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:41.486242 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:41.487200 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:41.487214 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:41.488136 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:41.489106 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.489131 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.490615 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.490620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.493075 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.493094 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.494243 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.494253 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.494522 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.495432 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:41.495456 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:41.499028 (+ 3572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:41.503432 (+ 4404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:41.503443 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:41.503462 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:41.504189 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:41.504708 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:41.504713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:41.504716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:41.505097 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:41.505894 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:41.505900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:41.506471 (+ 571us) client_negotiation.cc:769] Sending connection context
0621 06:52:41.506764 (+ 293us) client_negotiation.cc:240] Negotiation successful
0621 06:52:41.507411 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":379,"threads_started":1}
I20260621 06:52:41.524777 28878 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:37475
I20260621 06:52:41.524747 29012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:37475 every 8 connection(s)
I20260621 06:52:41.526120 28878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:52:41.528187 29013 heartbeater.cc:347] Connected to a master server at 127.20.120.254:39173
I20260621 06:52:41.528478 29013 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:41.529306 29013 heartbeater.cc:511] Master 127.20.120.254:39173 requested a full tablet report, sending...
I20260621 06:52:41.529577 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 28878
I20260621 06:52:41.529737 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:52:41.530386 20963 external_mini_cluster.cc:1428] Setting key cc38db6fc80c061ee28dd1c127e0a10b
I20260621 06:52:41.530961 28528 ts_manager.cc:194] Registered new tserver with Master: e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475)
I20260621 06:52:41.531044 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.516334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:39173 (local address 127.20.120.195:33879)
0621 06:52:41.517146 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:41.517162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:41.517327 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:41.517709 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:41.517717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:41.518488 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:41.519087 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.519101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.520448 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.520453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.522122 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.522144 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.522923 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.522931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.523099 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.523774 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:41.523805 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:41.524639 (+ 834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:41.528175 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:41.528181 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:41.528185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:41.528777 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:41.529186 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:41.529191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:41.529195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:41.529351 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:41.529983 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:41.529988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:41.530188 (+ 200us) client_negotiation.cc:769] Sending connection context
0621 06:52:41.530331 (+ 143us) client_negotiation.cc:240] Negotiation successful
0621 06:52:41.530630 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":494,"threads_started":1}
I20260621 06:52:41.531859 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.516190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:33879 (local address 127.20.120.254:39173)
0621 06:52:41.516374 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.516380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.517246 (+ 866us) server_negotiation.cc:407] Connection header received
0621 06:52:41.517481 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.517487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.517574 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.517679 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:41.519266 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.520246 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.522322 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.522725 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.522856 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.524829 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:41.524840 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:41.524843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:41.524873 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:41.528000 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:41.528892 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:41.528900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:41.528904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:41.529052 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:41.529495 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:41.529500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:41.529503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:41.529834 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:41.530007 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.531305 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.531519 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 06:52:41.532126 28528 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:42677
I20260621 06:52:41.533457 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 06:52:41 dist-test-slave-gmf2 krb5kdc[28483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 06:52:41 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:41 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 06:52:41.582518 28623 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:41.572439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.20.120.254:39173)
0621 06:52:41.572685 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.572692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.572914 (+ 222us) server_negotiation.cc:407] Connection header received
0621 06:52:41.573069 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.573073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.573139 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.573239 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:41.574656 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.575425 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.576613 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.577041 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.577143 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.582289 (+ 5146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35228: BlockingRecv error: recv got EOF from 127.0.0.1:35228 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 21 06:52:41 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:41.598501 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.585673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.20.120.254:39173)
0621 06:52:41.585846 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.585851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.586179 (+ 328us) server_negotiation.cc:407] Connection header received
0621 06:52:41.586336 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.586340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.586413 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.586506 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:41.587592 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.588281 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.589620 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.590069 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.590162 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.593836 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:41.593840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:41.593843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:41.593873 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:41.596268 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:41.596868 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:41.596872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:41.596874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:41.596982 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:41.597274 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:41.597277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:41.597278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:41.597603 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:41.597861 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.598077 (+ 216us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.598257 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 06:52:41.602604 28526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:52:41.609707 28526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:52:41.635766 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.622263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:37475 (local address 127.0.0.1:32936)
0621 06:52:41.623024 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:41.623081 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:41.623274 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:41.624997 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:41.625011 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:41.625054 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:41.625604 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.625620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.628718 (+ 3098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.628728 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.631958 (+ 3230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.632000 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.634618 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.634626 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.634899 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.634985 (+ 86us) client_negotiation.cc:769] Sending connection context
0621 06:52:41.635116 (+ 131us) client_negotiation.cc:240] Negotiation successful
0621 06:52:41.635337 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":295,"threads_started":1}
I20260621 06:52:41.638043 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.622628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.20.120.195:37475)
0621 06:52:41.624177 (+ 1549us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.624190 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.624230 (+ 40us) server_negotiation.cc:407] Connection header received
0621 06:52:41.624353 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.624377 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.624708 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.625876 (+ 1168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:41.626330 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.628516 (+ 2186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.632310 (+ 3794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.634339 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.636767 (+ 2428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.636986 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.637059 (+ 73us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.637234 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1348,"thread_start_us":1186,"threads_started":1}
I20260621 06:52:41.642493 28943 tablet_service.cc:1467] Processing CreateTablet for tablet 47bbbbf9ed3d4edbab4490546ab22bde (DEFAULT_TABLE table=test-table [id=46f775be3c00410b975cd89fca71c965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:41.643687 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.622935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:44587 (local address 127.0.0.1:33204)
0621 06:52:41.635207 (+ 12272us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:41.635225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:41.635435 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:41.636423 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:41.636427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:41.636449 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:41.636801 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.636811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.638568 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.638572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.640812 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.640823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.642903 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.642909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.643078 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.643110 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 06:52:41.643200 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 06:52:41.643333 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12134,"thread_start_us":15108,"threads_started":1}
I20260621 06:52:41.644168 28943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47bbbbf9ed3d4edbab4490546ab22bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:41.644419 29036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.623917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.20.120.193:38051)
0621 06:52:41.636923 (+ 13006us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.636936 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.636970 (+ 34us) server_negotiation.cc:407] Connection header received
0621 06:52:41.637088 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.637099 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.637393 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.637634 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:41.638402 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.639768 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.641910 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.643417 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.643595 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.643757 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.643962 (+ 205us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.644120 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12835,"thread_start_us":12685,"threads_started":1}
I20260621 06:52:41.644450 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.623378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.20.120.194:44587)
0621 06:52:41.634533 (+ 11155us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.634554 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.635314 (+ 760us) server_negotiation.cc:407] Connection header received
0621 06:52:41.635628 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.635636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.635974 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.636157 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:41.636987 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.638422 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.641045 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.642217 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.642592 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.642760 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.644028 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.644174 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10939,"thread_start_us":11183,"threads_started":1}
I20260621 06:52:41.644294 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.623369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38051 (local address 127.0.0.1:59474)
0621 06:52:41.636013 (+ 12644us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:41.636033 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:41.636124 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:41.637628 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:41.637633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:41.637657 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:41.637883 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.637897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.639906 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.639911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.641735 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.641748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.643588 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.643593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.643790 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.643819 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 06:52:41.643909 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 06:52:41.644038 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12556,"thread_start_us":11548,"threads_started":1}
I20260621 06:52:41.648157 28804 tablet_service.cc:1467] Processing CreateTablet for tablet 47bbbbf9ed3d4edbab4490546ab22bde (DEFAULT_TABLE table=test-table [id=46f775be3c00410b975cd89fca71c965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:41.648157 28665 tablet_service.cc:1467] Processing CreateTablet for tablet 47bbbbf9ed3d4edbab4490546ab22bde (DEFAULT_TABLE table=test-table [id=46f775be3c00410b975cd89fca71c965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:41.649555 28665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47bbbbf9ed3d4edbab4490546ab22bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:41.649570 28804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47bbbbf9ed3d4edbab4490546ab22bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:41.656924 29037 tablet_bootstrap.cc:492] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe: Bootstrap starting.
I20260621 06:52:41.660866 29037 tablet_bootstrap.cc:654] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:41.661979 29037 log.cc:824] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:41.664525 29039 tablet_bootstrap.cc:492] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3: Bootstrap starting.
I20260621 06:52:41.665323 29040 tablet_bootstrap.cc:492] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222: Bootstrap starting.
I20260621 06:52:41.666457 29037 tablet_bootstrap.cc:492] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe: No bootstrap required, opened a new log
I20260621 06:52:41.666801 29037 ts_tablet_manager.cc:1397] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260621 06:52:41.668964 29040 tablet_bootstrap.cc:654] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:41.670275 29040 log.cc:824] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:41.672235 29039 tablet_bootstrap.cc:654] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:41.673197 29039 log.cc:824] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:41.673496 29040 tablet_bootstrap.cc:492] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222: No bootstrap required, opened a new log
I20260621 06:52:41.673794 29040 ts_tablet_manager.cc:1397] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260621 06:52:41.676118 29039 tablet_bootstrap.cc:492] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3: No bootstrap required, opened a new log
I20260621 06:52:41.676409 29039 ts_tablet_manager.cc:1397] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260621 06:52:41.677565 29037 raft_consensus.cc:348] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } }
I20260621 06:52:41.678047 29037 raft_consensus.cc:374] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:41.678191 29037 raft_consensus.cc:729] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ad1765a77e44678de168473f3ccffe, State: Initialized, Role: FOLLOWER
I20260621 06:52:41.678829 29037 consensus_queue.cc:260] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } }
I20260621 06:52:41.679986 29013 heartbeater.cc:503] Master 127.20.120.254:39173 was elected leader, sending a full tablet report...
I20260621 06:52:41.680640 29037 ts_tablet_manager.cc:1428] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260621 06:52:41.683853 29040 raft_consensus.cc:348] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } }
I20260621 06:52:41.684363 29040 raft_consensus.cc:374] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:41.684480 29040 raft_consensus.cc:729] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f284c245134cc3b89c862385b1e222, State: Initialized, Role: FOLLOWER
I20260621 06:52:41.684639 29039 raft_consensus.cc:348] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } }
I20260621 06:52:41.685057 29039 raft_consensus.cc:374] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:41.685187 29039 raft_consensus.cc:729] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da34b0cd81c46138c244cffa93469d3, State: Initialized, Role: FOLLOWER
I20260621 06:52:41.685132 29040 consensus_queue.cc:260] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } }
I20260621 06:52:41.685810 29039 consensus_queue.cc:260] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } }
I20260621 06:52:41.687688 29039 ts_tablet_manager.cc:1428] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260621 06:52:41.688012 28874 heartbeater.cc:503] Master 127.20.120.254:39173 was elected leader, sending a full tablet report...
I20260621 06:52:41.688663 29040 ts_tablet_manager.cc:1428] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260621 06:52:41.701325 29044 raft_consensus.cc:482] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:52:41.701637 29044 raft_consensus.cc:504] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } }
I20260621 06:52:41.703378 29044 leader_election.cc:288] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475), 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193:38051)
I20260621 06:52:41.712191 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.703665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:45055 (local address 127.20.120.195:37475)
0621 06:52:41.703847 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.703851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.704510 (+ 659us) server_negotiation.cc:407] Connection header received
0621 06:52:41.704750 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.704758 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.704843 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.704908 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:41.705665 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.706736 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.709521 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.710811 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.711134 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.711223 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.711799 (+ 576us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.711935 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 06:52:41.712194 29036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.704585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:44657 (local address 127.20.120.193:38051)
0621 06:52:41.704803 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.704809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.705123 (+ 314us) server_negotiation.cc:407] Connection header received
0621 06:52:41.705265 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.705268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.705335 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.705418 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:41.705994 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.706958 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.709521 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.710828 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.711134 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.711223 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.711799 (+ 576us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.711933 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 06:52:41.712428 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.704410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38051 (local address 127.20.120.194:44657)
0621 06:52:41.705027 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:41.705044 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:41.705153 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:41.705468 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:41.705477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:41.705505 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:41.705841 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.705849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.707076 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.707082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.709358 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.709372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.711253 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.711260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.711578 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.711665 (+ 87us) client_negotiation.cc:769] Sending connection context
0621 06:52:41.711751 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 06:52:41.711891 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":138,"thread_start_us":295,"threads_started":1}
I20260621 06:52:41.712450 29046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.703705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:37475 (local address 127.20.120.194:45055)
0621 06:52:41.704349 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:41.704372 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:41.704535 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:41.705135 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:41.705141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:41.705160 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:41.705514 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.705526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.706861 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.706865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.709359 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.709372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.711257 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.711262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.711574 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.711650 (+ 76us) client_negotiation.cc:769] Sending connection context
0621 06:52:41.711785 (+ 135us) client_negotiation.cc:240] Negotiation successful
0621 06:52:41.711891 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":155,"thread_start_us":350,"threads_started":1}
I20260621 06:52:41.713500 28963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47bbbbf9ed3d4edbab4490546ab22bde" candidate_uuid: "f7f284c245134cc3b89c862385b1e222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ad1765a77e44678de168473f3ccffe" is_pre_election: true
I20260621 06:52:41.713541 28687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47bbbbf9ed3d4edbab4490546ab22bde" candidate_uuid: "f7f284c245134cc3b89c862385b1e222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da34b0cd81c46138c244cffa93469d3" is_pre_election: true
I20260621 06:52:41.714116 28963 raft_consensus.cc:2454] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7f284c245134cc3b89c862385b1e222 in term 0.
I20260621 06:52:41.714156 28687 raft_consensus.cc:2454] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7f284c245134cc3b89c862385b1e222 in term 0.
I20260621 06:52:41.714916 28758 leader_election.cc:302] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da34b0cd81c46138c244cffa93469d3, f7f284c245134cc3b89c862385b1e222; no voters:
I20260621 06:52:41.715467 29044 raft_consensus.cc:2781] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:52:41.715632 29044 raft_consensus.cc:482] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:52:41.715766 29044 raft_consensus.cc:3037] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:41.718120 29044 raft_consensus.cc:504] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } }
I20260621 06:52:41.718672 29044 leader_election.cc:288] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [CANDIDATE]: Term 1 election: Requested vote from peers e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475), 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193:38051)
I20260621 06:52:41.719156 28963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47bbbbf9ed3d4edbab4490546ab22bde" candidate_uuid: "f7f284c245134cc3b89c862385b1e222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ad1765a77e44678de168473f3ccffe"
I20260621 06:52:41.719255 28687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47bbbbf9ed3d4edbab4490546ab22bde" candidate_uuid: "f7f284c245134cc3b89c862385b1e222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da34b0cd81c46138c244cffa93469d3"
I20260621 06:52:41.719386 28963 raft_consensus.cc:3037] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:41.719467 28687 raft_consensus.cc:3037] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:41.722649 28687 raft_consensus.cc:2454] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f284c245134cc3b89c862385b1e222 in term 1.
I20260621 06:52:41.722649 28963 raft_consensus.cc:2454] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f284c245134cc3b89c862385b1e222 in term 1.
I20260621 06:52:41.723042 28758 leader_election.cc:302] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da34b0cd81c46138c244cffa93469d3, f7f284c245134cc3b89c862385b1e222; no voters:
I20260621 06:52:41.723362 29044 raft_consensus.cc:2781] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:41.723877 29044 raft_consensus.cc:686] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 1 LEADER]: Becoming Leader. State: Replica: f7f284c245134cc3b89c862385b1e222, State: Running, Role: LEADER
I20260621 06:52:41.724324 29044 consensus_queue.cc:237] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } }
I20260621 06:52:41.728971 28526 catalog_manager.cc:5495] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f284c245134cc3b89c862385b1e222 (127.20.120.194). New cstate: current_term: 1 leader_uuid: "f7f284c245134cc3b89c862385b1e222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } health_report { overall_health: HEALTHY } } }
W20260621 06:52:41.730057 28736 tablet.cc:2369] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:52:41.745592 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.740533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.20.120.194:44587)
0621 06:52:41.740702 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.740706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.740722 (+ 16us) server_negotiation.cc:407] Connection header received
0621 06:52:41.740907 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.740911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.740991 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.741157 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:52:41.741484 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.742333 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.743594 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.743982 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.744072 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.744425 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:52:41.745014 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:52:41.745120 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.745215 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.745277 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 06:52:41.753485 28687 raft_consensus.cc:1261] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Refusing update from remote peer f7f284c245134cc3b89c862385b1e222: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:41.753552 28963 raft_consensus.cc:1261] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Refusing update from remote peer f7f284c245134cc3b89c862385b1e222: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:41.754480 29048 consensus_queue.cc:1035] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:41.754802 29044 consensus_queue.cc:1035] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 06:52:41.762164 28875 tablet.cc:2369] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:52:41.769194 29051 mvcc.cc:205] Tried to move back new op lower bound from 7299173424133087232 to 7299173424026386432. Current Snapshot: MvccSnapshot[applied={T|T < 7299173424133087232}]
I20260621 06:52:41.769505 29053 mvcc.cc:205] Tried to move back new op lower bound from 7299173424133087232 to 7299173424026386432. Current Snapshot: MvccSnapshot[applied={T|T < 7299173424133087232 or (T in {7299173424133087232})}]
I20260621 06:52:41.780032 28527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:41.780349 28527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:41.784250 28527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb: Sending DeleteTablet for 3 replicas of tablet 47bbbbf9ed3d4edbab4490546ab22bde
I20260621 06:52:41.785308 28943 tablet_service.cc:1514] Processing DeleteTablet for tablet 47bbbbf9ed3d4edbab4490546ab22bde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:41 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:32936
I20260621 06:52:41.785355 28804 tablet_service.cc:1514] Processing DeleteTablet for tablet 47bbbbf9ed3d4edbab4490546ab22bde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:41 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:33204
I20260621 06:52:41.785564 28665 tablet_service.cc:1514] Processing DeleteTablet for tablet 47bbbbf9ed3d4edbab4490546ab22bde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:41 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:59474
I20260621 06:52:41.786868 28527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:52:41.788020 28527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:52:41.796711 29062 tablet_replica.cc:330] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3: stopping tablet replica
I20260621 06:52:41.797345 29062 raft_consensus.cc:2229] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:41.798055 29062 raft_consensus.cc:2258] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:41.798276 28803 tablet_service.cc:1467] Processing CreateTablet for tablet 016711738d3b4f4c84b52025872a63b1 (DEFAULT_TABLE table=test-table [id=2af6f641503048cdb642388351d4f71c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:41.798622 29060 tablet_replica.cc:330] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe: stopping tablet replica
I20260621 06:52:41.799036 28803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016711738d3b4f4c84b52025872a63b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:41.799034 29060 raft_consensus.cc:2229] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:41.799480 29060 raft_consensus.cc:2258] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:41.801776 29060 ts_tablet_manager.cc:1905] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:41.804080 28665 tablet_service.cc:1467] Processing CreateTablet for tablet 016711738d3b4f4c84b52025872a63b1 (DEFAULT_TABLE table=test-table [id=2af6f641503048cdb642388351d4f71c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:41.804659 28665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016711738d3b4f4c84b52025872a63b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:41.808861 29060 ts_tablet_manager.cc:1918] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:41.809103 29060 log.cc:1196] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/47bbbbf9ed3d4edbab4490546ab22bde
I20260621 06:52:41.809743 29060 ts_tablet_manager.cc:1939] T 47bbbbf9ed3d4edbab4490546ab22bde P e3ad1765a77e44678de168473f3ccffe: Deleting consensus metadata
I20260621 06:52:41.810772 28943 tablet_service.cc:1467] Processing CreateTablet for tablet 016711738d3b4f4c84b52025872a63b1 (DEFAULT_TABLE table=test-table [id=2af6f641503048cdb642388351d4f71c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:41.811208 28943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016711738d3b4f4c84b52025872a63b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:41.812232 28515 catalog_manager.cc:4841] TS e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475): tablet 47bbbbf9ed3d4edbab4490546ab22bde (table test-table [id=46f775be3c00410b975cd89fca71c965]) successfully deleted
I20260621 06:52:41.816171 29062 ts_tablet_manager.cc:1905] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:41.817679 29037 tablet_bootstrap.cc:492] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe: Bootstrap starting.
I20260621 06:52:41.819383 29037 tablet_bootstrap.cc:654] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:41.827986 29039 tablet_bootstrap.cc:492] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3: Bootstrap starting.
I20260621 06:52:41.830247 29039 tablet_bootstrap.cc:654] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:41.834822 29040 tablet_bootstrap.cc:492] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222: Bootstrap starting.
I20260621 06:52:41.835112 29037 tablet_bootstrap.cc:492] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe: No bootstrap required, opened a new log
I20260621 06:52:41.835258 29037 ts_tablet_manager.cc:1397] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20260621 06:52:41.837212 29040 tablet_bootstrap.cc:654] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:41.837106 29037 raft_consensus.cc:348] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:41.837343 29037 raft_consensus.cc:374] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:41.837453 29037 raft_consensus.cc:729] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ad1765a77e44678de168473f3ccffe, State: Initialized, Role: FOLLOWER
I20260621 06:52:41.837764 29037 consensus_queue.cc:260] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:41.841723 29037 ts_tablet_manager.cc:1428] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260621 06:52:41.850030 29062 ts_tablet_manager.cc:1918] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:41.850222 29062 log.cc:1196] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/47bbbbf9ed3d4edbab4490546ab22bde
I20260621 06:52:41.853013 29062 ts_tablet_manager.cc:1939] T 47bbbbf9ed3d4edbab4490546ab22bde P 9da34b0cd81c46138c244cffa93469d3: Deleting consensus metadata
I20260621 06:52:41.854959 28515 catalog_manager.cc:4841] TS 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193:38051): tablet 47bbbbf9ed3d4edbab4490546ab22bde (table test-table [id=46f775be3c00410b975cd89fca71c965]) successfully deleted
I20260621 06:52:41.855031 29039 tablet_bootstrap.cc:492] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3: No bootstrap required, opened a new log
I20260621 06:52:41.855173 29039 ts_tablet_manager.cc:1397] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.009s
I20260621 06:52:41.855204 29061 tablet_replica.cc:330] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222: stopping tablet replica
I20260621 06:52:41.855831 29040 tablet_bootstrap.cc:492] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222: No bootstrap required, opened a new log
I20260621 06:52:41.855865 29061 raft_consensus.cc:2229] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:52:41.856122 29040 ts_tablet_manager.cc:1397] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.010s
I20260621 06:52:41.855993 29039 raft_consensus.cc:348] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:41.856271 29039 raft_consensus.cc:374] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:41.856395 29039 raft_consensus.cc:729] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da34b0cd81c46138c244cffa93469d3, State: Initialized, Role: FOLLOWER
I20260621 06:52:41.856684 29061 raft_consensus.cc:2258] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:41.856735 29039 consensus_queue.cc:260] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:41.857339 29039 ts_tablet_manager.cc:1428] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260621 06:52:41.857832 29040 raft_consensus.cc:348] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:41.858139 29040 raft_consensus.cc:374] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:41.858289 29040 raft_consensus.cc:729] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f284c245134cc3b89c862385b1e222, State: Initialized, Role: FOLLOWER
I20260621 06:52:41.858640 29040 consensus_queue.cc:260] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:41.859324 29040 ts_tablet_manager.cc:1428] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 06:52:41.861555 29061 ts_tablet_manager.cc:1905] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:41.868892 29061 ts_tablet_manager.cc:1918] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:41.869074 29061 log.cc:1196] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/47bbbbf9ed3d4edbab4490546ab22bde
I20260621 06:52:41.869621 29061 ts_tablet_manager.cc:1939] T 47bbbbf9ed3d4edbab4490546ab22bde P f7f284c245134cc3b89c862385b1e222: Deleting consensus metadata
I20260621 06:52:41.871430 28513 catalog_manager.cc:4841] TS f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587): tablet 47bbbbf9ed3d4edbab4490546ab22bde (table test-table [id=46f775be3c00410b975cd89fca71c965]) successfully deleted
I20260621 06:52:41.916219 29045 raft_consensus.cc:482] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:52:41.916478 29045 raft_consensus.cc:504] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:41.917372 29043 raft_consensus.cc:482] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:52:41.917590 29043 raft_consensus.cc:504] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:41.918416 29045 leader_election.cc:288] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587), e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475)
I20260621 06:52:41.919317 29043 leader_election.cc:288] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193:38051), f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587)
I20260621 06:52:41.928251 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.918961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:58213 (local address 127.20.120.194:44587)
0621 06:52:41.919138 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.919151 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.920049 (+ 898us) server_negotiation.cc:407] Connection header received
0621 06:52:41.920349 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.920369 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.920451 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.920508 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:41.921577 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.922809 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.925421 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.926692 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.926819 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.926905 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.927739 (+ 834us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.927878 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 06:52:41.929189 29067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.918625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:44587 (local address 127.20.120.193:58213)
0621 06:52:41.919916 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:41.919939 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:41.920170 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:41.920726 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:41.920733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:41.920755 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:41.921370 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.921383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.922982 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.922989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.925241 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.925254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.927362 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.927368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.927556 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.927588 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 06:52:41.928624 (+ 1036us) client_negotiation.cc:240] Negotiation successful
0621 06:52:41.928766 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1148,"mutex_wait_us":25,"thread_start_us":2133,"threads_started":1}
I20260621 06:52:41.930399 29036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.920992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:58099 (local address 127.20.120.193:38051)
0621 06:52:41.921148 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.921152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.921168 (+ 16us) server_negotiation.cc:407] Connection header received
0621 06:52:41.921259 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.921264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.921338 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.921440 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:41.921980 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.923130 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.925663 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.927091 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.927244 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.927313 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.928995 (+ 1682us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.929130 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 06:52:41.930390 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.920625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:34745 (local address 127.20.120.194:44587)
0621 06:52:41.922287 (+ 1662us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.922293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.922308 (+ 15us) server_negotiation.cc:407] Connection header received
0621 06:52:41.922356 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.922361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.922419 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.922504 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:41.923338 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.924628 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.926973 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.928321 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.928570 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.928705 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.929854 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.930003 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1464,"thread_start_us":4901,"threads_started":1}
I20260621 06:52:41.930785 28902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.919503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38051 (local address 127.20.120.195:58099)
0621 06:52:41.920131 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:41.920147 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:41.920255 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:41.921574 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:41.921579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:41.921603 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:41.921818 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.921828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.923336 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.923343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.925381 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.925396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.927870 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.927876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.928095 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.928130 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 06:52:41.928221 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 06:52:41.929219 (+ 998us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"spinlock_wait_cycles":45440}
I20260621 06:52:41.931069 28824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016711738d3b4f4c84b52025872a63b1" candidate_uuid: "9da34b0cd81c46138c244cffa93469d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f284c245134cc3b89c862385b1e222" is_pre_election: true
I20260621 06:52:41.931597 28824 raft_consensus.cc:2454] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da34b0cd81c46138c244cffa93469d3 in term 0.
I20260621 06:52:41.931605 28687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016711738d3b4f4c84b52025872a63b1" candidate_uuid: "e3ad1765a77e44678de168473f3ccffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da34b0cd81c46138c244cffa93469d3" is_pre_election: true
I20260621 06:52:41.931862 28687 raft_consensus.cc:2454] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3ad1765a77e44678de168473f3ccffe in term 0.
I20260621 06:52:41.932476 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.919831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:44587 (local address 127.20.120.195:34745)
0621 06:52:41.921564 (+ 1733us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:41.921585 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:41.921712 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:41.922738 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:41.922742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:41.922763 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:41.923143 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.923156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.924795 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.924800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.926811 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.926823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.928866 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.928875 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.929047 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.929078 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 06:52:41.929183 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 06:52:41.929328 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"spinlock_wait_cycles":3840,"thread_start_us":6125,"threads_started":1}
I20260621 06:52:41.932389 28616 leader_election.cc:302] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da34b0cd81c46138c244cffa93469d3, f7f284c245134cc3b89c862385b1e222; no voters:
I20260621 06:52:41.932842 29045 raft_consensus.cc:2781] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:52:41.932968 29045 raft_consensus.cc:482] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:52:41.933063 29045 raft_consensus.cc:3037] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:41.932978 28897 leader_election.cc:302] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da34b0cd81c46138c244cffa93469d3, e3ad1765a77e44678de168473f3ccffe; no voters:
I20260621 06:52:41.933375 29043 raft_consensus.cc:2781] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:52:41.933476 29043 raft_consensus.cc:482] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:52:41.933563 29043 raft_consensus.cc:3037] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:41.934329 28824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016711738d3b4f4c84b52025872a63b1" candidate_uuid: "e3ad1765a77e44678de168473f3ccffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f284c245134cc3b89c862385b1e222" is_pre_election: true
I20260621 06:52:41.934552 28824 raft_consensus.cc:2454] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3ad1765a77e44678de168473f3ccffe in term 0.
I20260621 06:52:41.936218 29045 raft_consensus.cc:504] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:41.936530 29043 raft_consensus.cc:504] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:41.937494 28687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016711738d3b4f4c84b52025872a63b1" candidate_uuid: "e3ad1765a77e44678de168473f3ccffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da34b0cd81c46138c244cffa93469d3"
I20260621 06:52:41.937474 28824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016711738d3b4f4c84b52025872a63b1" candidate_uuid: "9da34b0cd81c46138c244cffa93469d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f284c245134cc3b89c862385b1e222"
I20260621 06:52:41.937647 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.919410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:52345 (local address 127.20.120.195:37475)
0621 06:52:41.919615 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.919623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.920258 (+ 635us) server_negotiation.cc:407] Connection header received
0621 06:52:41.920474 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.920481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.920544 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.920596 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:52:41.929644 (+ 9048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.931111 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.933804 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.936533 (+ 2729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.936682 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.936748 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:41.937279 (+ 531us) server_negotiation.cc:299] Negotiation successful
0621 06:52:41.937412 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1423,"server-negotiator.queue_time_us":79}
I20260621 06:52:41.937661 28824 raft_consensus.cc:3037] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:41.937736 28687 raft_consensus.cc:2379] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e3ad1765a77e44678de168473f3ccffe in current term 1: Already voted for candidate 9da34b0cd81c46138c244cffa93469d3 in this term.
I20260621 06:52:41.938325 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.918879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:37475 (local address 127.20.120.193:52345)
0621 06:52:41.920181 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:41.920196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:41.920331 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:41.920748 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:41.920751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:41.920766 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:52:41.921353 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.921370 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.931297 (+ 9927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.931303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.933620 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.933638 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.936894 (+ 3256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.936900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.937111 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.937142 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 06:52:41.937830 (+ 688us) client_negotiation.cc:240] Negotiation successful
0621 06:52:41.937984 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1203,"mutex_wait_us":66,"thread_start_us":786,"threads_started":1}
I20260621 06:52:41.938689 29043 leader_election.cc:288] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [CANDIDATE]: Term 1 election: Requested vote from peers 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193:38051), f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587)
I20260621 06:52:41.939009 29045 leader_election.cc:288] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [CANDIDATE]: Term 1 election: Requested vote from peers f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587), e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475)
I20260621 06:52:41.939038 28823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016711738d3b4f4c84b52025872a63b1" candidate_uuid: "e3ad1765a77e44678de168473f3ccffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f284c245134cc3b89c862385b1e222"
I20260621 06:52:41.939587 28963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016711738d3b4f4c84b52025872a63b1" candidate_uuid: "9da34b0cd81c46138c244cffa93469d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ad1765a77e44678de168473f3ccffe" is_pre_election: true
I20260621 06:52:41.939913 28963 raft_consensus.cc:2379] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da34b0cd81c46138c244cffa93469d3 in current term 1: Already voted for candidate e3ad1765a77e44678de168473f3ccffe in this term.
I20260621 06:52:41.939857 28962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016711738d3b4f4c84b52025872a63b1" candidate_uuid: "9da34b0cd81c46138c244cffa93469d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ad1765a77e44678de168473f3ccffe"
I20260621 06:52:41.939918 28895 leader_election.cc:302] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3ad1765a77e44678de168473f3ccffe; no voters: 9da34b0cd81c46138c244cffa93469d3, f7f284c245134cc3b89c862385b1e222
I20260621 06:52:41.940169 28962 raft_consensus.cc:2379] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da34b0cd81c46138c244cffa93469d3 in current term 1: Already voted for candidate e3ad1765a77e44678de168473f3ccffe in this term.
I20260621 06:52:41.940439 29043 raft_consensus.cc:2726] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260621 06:52:41.941756 28824 raft_consensus.cc:2454] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da34b0cd81c46138c244cffa93469d3 in term 1.
I20260621 06:52:41.942099 28616 leader_election.cc:302] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9da34b0cd81c46138c244cffa93469d3, f7f284c245134cc3b89c862385b1e222; no voters: e3ad1765a77e44678de168473f3ccffe
I20260621 06:52:41.942394 29045 raft_consensus.cc:2781] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:41.942879 29045 raft_consensus.cc:686] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 1 LEADER]: Becoming Leader. State: Replica: 9da34b0cd81c46138c244cffa93469d3, State: Running, Role: LEADER
I20260621 06:52:41.943274 29045 consensus_queue.cc:237] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:41.946885 28526 catalog_manager.cc:5495] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193). New cstate: current_term: 1 leader_uuid: "9da34b0cd81c46138c244cffa93469d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } health_report { overall_health: UNKNOWN } } }
Jun 21 06:52:41 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024759, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.254@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:42.003549 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.988669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:39173 (local address 127.0.0.1:35238)
0621 06:52:41.989316 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:41.989332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:41.989467 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:41.989802 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:41.989805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:41.990514 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:41.990759 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.990767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.992107 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.992110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.993034 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:41.993044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.994129 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.994134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:41.994252 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.994948 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:41.994963 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:41.997357 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:42.000976 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:42.000987 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:42.000993 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:42.001403 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:42.001762 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:42.001765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:42.001768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:42.001896 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:42.002584 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:42.002595 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:42.002810 (+ 215us) client_negotiation.cc:769] Sending connection context
0621 06:52:42.002989 (+ 179us) client_negotiation.cc:240] Negotiation successful
0621 06:52:42.003206 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":368,"threads_started":1}
I20260621 06:52:42.003795 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:41.988567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.20.120.254:39173)
0621 06:52:41.988818 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:41.988822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:41.989420 (+ 598us) server_negotiation.cc:407] Connection header received
0621 06:52:41.989577 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:41.989583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:41.989663 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:41.989773 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:41.990906 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.991952 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.993169 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:41.993958 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:41.994075 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:41.997547 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:41.997554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:41.997559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:41.997594 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:42.000836 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:42.001514 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:42.001520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:42.001524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:42.001658 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:42.002036 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:42.002042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:42.002045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:42.002417 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:42.002581 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:42.003104 (+ 523us) server_negotiation.cc:299] Negotiation successful
0621 06:52:42.003349 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":95}
I20260621 06:52:42.009593 28526 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:35238:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 06:52:42.010879 28526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:52:42.018491 28804 tablet_service.cc:1467] Processing CreateTablet for tablet fd69bd7cafef45edb633766ff2df0d7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aef154b9bfc247748c97605f341d57f9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 06:52:42.019086 28804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd69bd7cafef45edb633766ff2df0d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:42.019047 28943 tablet_service.cc:1467] Processing CreateTablet for tablet fd69bd7cafef45edb633766ff2df0d7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aef154b9bfc247748c97605f341d57f9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 06:52:42.019068 28665 tablet_service.cc:1467] Processing CreateTablet for tablet fd69bd7cafef45edb633766ff2df0d7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aef154b9bfc247748c97605f341d57f9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 06:52:42.019552 28943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd69bd7cafef45edb633766ff2df0d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:42.019593 28665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd69bd7cafef45edb633766ff2df0d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:42.026393 29040 tablet_bootstrap.cc:492] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222: Bootstrap starting.
I20260621 06:52:42.026886 29037 tablet_bootstrap.cc:492] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe: Bootstrap starting.
I20260621 06:52:42.028142 29040 tablet_bootstrap.cc:654] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:42.028510 29037 tablet_bootstrap.cc:654] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe: Neither blocks nor log segments found. Creating new log.
W20260621 06:52:42.028877 29014 tablet.cc:2369] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:52:42.031876 29040 tablet_bootstrap.cc:492] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222: No bootstrap required, opened a new log
I20260621 06:52:42.032127 29039 tablet_bootstrap.cc:492] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: Bootstrap starting.
I20260621 06:52:42.032127 29040 ts_tablet_manager.cc:1397] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260621 06:52:42.032838 29040 raft_consensus.cc:348] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.033077 29040 raft_consensus.cc:374] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:42.033142 29040 raft_consensus.cc:729] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f284c245134cc3b89c862385b1e222, State: Initialized, Role: FOLLOWER
I20260621 06:52:42.033674 29039 tablet_bootstrap.cc:654] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:42.034271 29040 consensus_queue.cc:260] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.034690 29048 tablet_replica.cc:439] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } }
I20260621 06:52:42.034991 29048 tablet_replica.cc:442] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:52:42.038585 29040 ts_tablet_manager.cc:1428] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260621 06:52:42.043497 29039 tablet_bootstrap.cc:492] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: No bootstrap required, opened a new log
I20260621 06:52:42.043730 29039 ts_tablet_manager.cc:1397] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.006s
I20260621 06:52:42.044167 29037 tablet_bootstrap.cc:492] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe: No bootstrap required, opened a new log
I20260621 06:52:42.044412 29037 ts_tablet_manager.cc:1397] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.004s
I20260621 06:52:42.044449 29039 raft_consensus.cc:348] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.044724 29039 raft_consensus.cc:374] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:42.044835 29039 raft_consensus.cc:729] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da34b0cd81c46138c244cffa93469d3, State: Initialized, Role: FOLLOWER
I20260621 06:52:42.045116 29039 consensus_queue.cc:260] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.045679 29039 ts_tablet_manager.cc:1428] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260621 06:52:42.045570 29045 tablet_replica.cc:439] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } }
I20260621 06:52:42.045842 29045 tablet_replica.cc:442] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:52:42.045288 29037 raft_consensus.cc:348] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.046779 29037 raft_consensus.cc:374] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:42.046895 29037 raft_consensus.cc:729] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ad1765a77e44678de168473f3ccffe, State: Initialized, Role: FOLLOWER
I20260621 06:52:42.047178 29037 consensus_queue.cc:260] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.047667 29043 tablet_replica.cc:439] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } }
I20260621 06:52:42.047956 29043 tablet_replica.cc:442] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:52:42.048199 29037 ts_tablet_manager.cc:1428] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260621 06:52:42.087898 29045 raft_consensus.cc:482] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:52:42.088146 29045 raft_consensus.cc:504] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.088891 29045 leader_election.cc:288] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587), e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475)
I20260621 06:52:42.089424 28824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd69bd7cafef45edb633766ff2df0d7c" candidate_uuid: "9da34b0cd81c46138c244cffa93469d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f284c245134cc3b89c862385b1e222" is_pre_election: true
I20260621 06:52:42.089510 28962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd69bd7cafef45edb633766ff2df0d7c" candidate_uuid: "9da34b0cd81c46138c244cffa93469d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ad1765a77e44678de168473f3ccffe" is_pre_election: true
I20260621 06:52:42.089704 28824 raft_consensus.cc:2454] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da34b0cd81c46138c244cffa93469d3 in term 0.
I20260621 06:52:42.089769 28962 raft_consensus.cc:2454] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da34b0cd81c46138c244cffa93469d3 in term 0.
I20260621 06:52:42.090193 28618 leader_election.cc:302] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da34b0cd81c46138c244cffa93469d3, e3ad1765a77e44678de168473f3ccffe; no voters:
I20260621 06:52:42.090546 29045 raft_consensus.cc:2781] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:52:42.090651 29045 raft_consensus.cc:482] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:52:42.090781 29045 raft_consensus.cc:3037] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:42.092768 29045 raft_consensus.cc:504] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.093389 29045 leader_election.cc:288] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [CANDIDATE]: Term 1 election: Requested vote from peers f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587), e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475)
I20260621 06:52:42.093845 28824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd69bd7cafef45edb633766ff2df0d7c" candidate_uuid: "9da34b0cd81c46138c244cffa93469d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f284c245134cc3b89c862385b1e222"
I20260621 06:52:42.093881 28962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd69bd7cafef45edb633766ff2df0d7c" candidate_uuid: "9da34b0cd81c46138c244cffa93469d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ad1765a77e44678de168473f3ccffe"
I20260621 06:52:42.094125 28962 raft_consensus.cc:3037] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:42.094141 28824 raft_consensus.cc:3037] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:42.097168 28962 raft_consensus.cc:2454] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da34b0cd81c46138c244cffa93469d3 in term 1.
I20260621 06:52:42.097160 28824 raft_consensus.cc:2454] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da34b0cd81c46138c244cffa93469d3 in term 1.
I20260621 06:52:42.097671 28616 leader_election.cc:302] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da34b0cd81c46138c244cffa93469d3, f7f284c245134cc3b89c862385b1e222; no voters:
I20260621 06:52:42.098014 29045 raft_consensus.cc:2781] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:42.098210 29045 raft_consensus.cc:686] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [term 1 LEADER]: Becoming Leader. State: Replica: 9da34b0cd81c46138c244cffa93469d3, State: Running, Role: LEADER
I20260621 06:52:42.098476 29045 consensus_queue.cc:237] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.100498 29070 tablet_replica.cc:439] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: TxnStatusTablet state changed. Reason: New leader 9da34b0cd81c46138c244cffa93469d3. Latest consensus state: current_term: 1 leader_uuid: "9da34b0cd81c46138c244cffa93469d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } }
I20260621 06:52:42.100895 29070 tablet_replica.cc:442] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: This TxnStatusTablet replica's current role is: LEADER
I20260621 06:52:42.101687 29082 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 06:52:42.101938 29082 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 06:52:42.102440 28527 catalog_manager.cc:5495] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193). New cstate: current_term: 1 leader_uuid: "9da34b0cd81c46138c244cffa93469d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } health_report { overall_health: UNKNOWN } } }
I20260621 06:52:42.137133 29036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:42.131916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.20.120.193:38051)
0621 06:52:42.132124 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:42.132128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:42.132142 (+ 14us) server_negotiation.cc:407] Connection header received
0621 06:52:42.132218 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:42.132222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:42.132293 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:42.132438 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:52:42.132786 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.133577 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.134943 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.135370 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.135494 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:42.135942 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:52:42.136466 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:52:42.136616 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:42.136694 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 06:52:42.136775 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260621 06:52:42.137109 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:42.131467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38051 (local address 127.0.0.1:59476)
0621 06:52:42.131698 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:42.131721 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:42.131865 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:42.132453 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:42.132466 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:42.132478 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 06:52:42.132653 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:42.132661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.133706 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.133712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:42.134795 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:42.134806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.135524 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.135529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:42.135686 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:42.135737 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 06:52:42.136575 (+ 838us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 06:52:42.136594 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 06:52:42.136676 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 06:52:42.136821 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 06:52:42.141500 28962 raft_consensus.cc:1261] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Refusing update from remote peer 9da34b0cd81c46138c244cffa93469d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:42.142076 28824 raft_consensus.cc:1261] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Refusing update from remote peer 9da34b0cd81c46138c244cffa93469d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:42.142839 29045 consensus_queue.cc:1035] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:42.143206 29070 consensus_queue.cc:1035] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:42.148213 29043 tablet_replica.cc:439] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe: TxnStatusTablet state changed. Reason: New leader 9da34b0cd81c46138c244cffa93469d3. Latest consensus state: current_term: 1 leader_uuid: "9da34b0cd81c46138c244cffa93469d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } }
I20260621 06:52:42.148500 29043 tablet_replica.cc:442] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:52:42.148805 29070 tablet_replica.cc:439] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9da34b0cd81c46138c244cffa93469d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } }
I20260621 06:52:42.149049 29070 tablet_replica.cc:442] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: This TxnStatusTablet replica's current role is: LEADER
I20260621 06:52:42.152806 29043 tablet_replica.cc:439] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9da34b0cd81c46138c244cffa93469d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } }
I20260621 06:52:42.152921 29052 mvcc.cc:205] Tried to move back new op lower bound from 7299173425724547072 to 7299173425557725184. Current Snapshot: MvccSnapshot[applied={T|T < 7299173425724547072}]
I20260621 06:52:42.153069 29043 tablet_replica.cc:442] T fd69bd7cafef45edb633766ff2df0d7c P e3ad1765a77e44678de168473f3ccffe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:52:42.163625 29036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:42.156683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.20.120.193:38051)
0621 06:52:42.156872 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:42.156879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:42.156897 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:52:42.156959 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:42.156964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:42.157054 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:42.157641 (+ 587us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:52:42.157779 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.158903 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.160885 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.161576 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.162227 (+ 651us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:42.162360 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:52:42.162495 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:52:42.163138 (+ 643us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:42.163193 (+ 55us) server_negotiation.cc:299] Negotiation successful
0621 06:52:42.163280 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 06:52:42.164896 29048 tablet_replica.cc:439] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222: TxnStatusTablet state changed. Reason: New leader 9da34b0cd81c46138c244cffa93469d3. Latest consensus state: current_term: 1 leader_uuid: "9da34b0cd81c46138c244cffa93469d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } }
I20260621 06:52:42.165248 29048 tablet_replica.cc:442] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:52:42.169441 29045 tablet_replica.cc:439] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9da34b0cd81c46138c244cffa93469d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } }
I20260621 06:52:42.169785 29045 tablet_replica.cc:442] T fd69bd7cafef45edb633766ff2df0d7c P 9da34b0cd81c46138c244cffa93469d3: This TxnStatusTablet replica's current role is: LEADER
I20260621 06:52:42.172053 29048 tablet_replica.cc:439] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9da34b0cd81c46138c244cffa93469d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } } }
I20260621 06:52:42.172345 29048 tablet_replica.cc:442] T fd69bd7cafef45edb633766ff2df0d7c P f7f284c245134cc3b89c862385b1e222: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 06:52:42.192688 29036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:42.185020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:58997 (local address 127.20.120.193:38051)
0621 06:52:42.185201 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:42.185207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:42.185856 (+ 649us) server_negotiation.cc:407] Connection header received
0621 06:52:42.186117 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:42.186123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:42.186213 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:42.186309 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:52:42.186973 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.188097 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.190451 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.191110 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.191238 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:42.191784 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:52:42.191942 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:52:42.192072 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:42.192249 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 06:52:42.192381 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":68}
I20260621 06:52:42.193110 29089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:42.185071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38051 (local address 127.20.120.193:58997)
0621 06:52:42.185736 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:42.185755 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:42.185976 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:42.186345 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:42.186350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:42.186363 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 06:52:42.186758 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:42.186776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.188236 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.188241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:42.190293 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:42.190310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.191320 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.191327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:42.191514 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:42.191584 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 06:52:42.192098 (+ 514us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 06:52:42.192111 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 06:52:42.192210 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 06:52:42.192413 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":1,"thread_start_us":335,"threads_started":1}
I20260621 06:52:42.205336 28824 raft_consensus.cc:1261] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Refusing update from remote peer 9da34b0cd81c46138c244cffa93469d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:42.205349 28962 raft_consensus.cc:1261] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Refusing update from remote peer 9da34b0cd81c46138c244cffa93469d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:42.205947 29070 consensus_queue.cc:1035] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:42.206225 29045 consensus_queue.cc:1035] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:42.308468 28527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:42.308730 28527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:42.312088 28527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb: Sending DeleteTablet for 3 replicas of tablet 016711738d3b4f4c84b52025872a63b1
I20260621 06:52:42.313274 28943 tablet_service.cc:1514] Processing DeleteTablet for tablet 016711738d3b4f4c84b52025872a63b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:42 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:32936
I20260621 06:52:42.313480 28664 tablet_service.cc:1514] Processing DeleteTablet for tablet 016711738d3b4f4c84b52025872a63b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:42 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:59474
I20260621 06:52:42.313705 28804 tablet_service.cc:1514] Processing DeleteTablet for tablet 016711738d3b4f4c84b52025872a63b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:42 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:33204
I20260621 06:52:42.314011 29062 tablet_replica.cc:330] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3: stopping tablet replica
I20260621 06:52:42.314251 29062 raft_consensus.cc:2229] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:52:42.314711 29062 raft_consensus.cc:2258] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:42.314987 29061 tablet_replica.cc:330] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222: stopping tablet replica
I20260621 06:52:42.315176 29061 raft_consensus.cc:2229] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:42.315348 29061 raft_consensus.cc:2258] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:42.317023 29061 ts_tablet_manager.cc:1905] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:42.323024 29062 ts_tablet_manager.cc:1905] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:42.332767 29061 ts_tablet_manager.cc:1918] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:52:42.334717 29062 ts_tablet_manager.cc:1918] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:52:42.334947 29062 log.cc:1196] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/016711738d3b4f4c84b52025872a63b1
I20260621 06:52:42.335186 29061 log.cc:1196] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/016711738d3b4f4c84b52025872a63b1
I20260621 06:52:42.335656 29062 ts_tablet_manager.cc:1939] T 016711738d3b4f4c84b52025872a63b1 P 9da34b0cd81c46138c244cffa93469d3: Deleting consensus metadata
I20260621 06:52:42.336133 29061 ts_tablet_manager.cc:1939] T 016711738d3b4f4c84b52025872a63b1 P f7f284c245134cc3b89c862385b1e222: Deleting consensus metadata
I20260621 06:52:42.337450 28515 catalog_manager.cc:4841] TS 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193:38051): tablet 016711738d3b4f4c84b52025872a63b1 (table test-table [id=2af6f641503048cdb642388351d4f71c]) successfully deleted
I20260621 06:52:42.338145 28513 catalog_manager.cc:4841] TS f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587): tablet 016711738d3b4f4c84b52025872a63b1 (table test-table [id=2af6f641503048cdb642388351d4f71c]) successfully deleted
I20260621 06:52:42.341449 29097 tablet_replica.cc:330] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe: stopping tablet replica
I20260621 06:52:42.341646 29097 raft_consensus.cc:2229] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:42.341981 29097 raft_consensus.cc:2258] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:42.343304 29097 ts_tablet_manager.cc:1905] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:42.349021 29097 ts_tablet_manager.cc:1918] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:52:42.349201 29097 log.cc:1196] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/016711738d3b4f4c84b52025872a63b1
I20260621 06:52:42.349560 29097 ts_tablet_manager.cc:1939] T 016711738d3b4f4c84b52025872a63b1 P e3ad1765a77e44678de168473f3ccffe: Deleting consensus metadata
I20260621 06:52:42.350692 28515 catalog_manager.cc:4841] TS e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475): tablet 016711738d3b4f4c84b52025872a63b1 (table test-table [id=2af6f641503048cdb642388351d4f71c]) successfully deleted
Jun 21 06:52:42 dist-test-slave-gmf2 krb5kdc[28483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 06:52:42 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:42 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 06:52:42.393234 28623 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:42.380711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.20.120.254:39173)
0621 06:52:42.380904 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:42.380912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:42.381265 (+ 353us) server_negotiation.cc:407] Connection header received
0621 06:52:42.381474 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:42.381478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:42.381543 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:42.381648 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:42.382944 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.383642 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.384848 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.385242 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.385384 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:42.392940 (+ 7556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35246: BlockingRecv error: recv got EOF from 127.0.0.1:35246 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 21 06:52:42 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM
I20260621 06:52:42.410460 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:42.396087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.20.120.254:39173)
0621 06:52:42.396266 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:42.396272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:42.396555 (+ 283us) server_negotiation.cc:407] Connection header received
0621 06:52:42.396688 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:42.396693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:42.396781 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:42.396885 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:42.398063 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.399131 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.400329 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.400947 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.401086 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:42.405211 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:42.405216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:42.405219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:42.405253 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:42.407885 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:42.408656 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:42.408659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:42.408662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:42.408774 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:42.409151 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:42.409154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:42.409156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:42.409415 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:42.409625 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:42.409963 (+ 338us) server_negotiation.cc:299] Negotiation successful
0621 06:52:42.410136 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 06:52:42.414127 28527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:52:42.414822 28527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:52:42.421345 28664 tablet_service.cc:1467] Processing CreateTablet for tablet 38263fae16e1404f85b68adf9b37206f (DEFAULT_TABLE table=test-table [id=8ccbd0a5e476426b8fc29cd46810560c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:42.421557 28804 tablet_service.cc:1467] Processing CreateTablet for tablet 38263fae16e1404f85b68adf9b37206f (DEFAULT_TABLE table=test-table [id=8ccbd0a5e476426b8fc29cd46810560c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:42.421872 28664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38263fae16e1404f85b68adf9b37206f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:42.422037 28804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38263fae16e1404f85b68adf9b37206f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:42.426065 29040 tablet_bootstrap.cc:492] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222: Bootstrap starting.
I20260621 06:52:42.426913 28943 tablet_service.cc:1467] Processing CreateTablet for tablet 38263fae16e1404f85b68adf9b37206f (DEFAULT_TABLE table=test-table [id=8ccbd0a5e476426b8fc29cd46810560c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:42.427353 28943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38263fae16e1404f85b68adf9b37206f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:42.427587 29040 tablet_bootstrap.cc:654] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:42.433081 29040 tablet_bootstrap.cc:492] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222: No bootstrap required, opened a new log
I20260621 06:52:42.433104 29037 tablet_bootstrap.cc:492] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe: Bootstrap starting.
I20260621 06:52:42.433250 29040 ts_tablet_manager.cc:1397] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260621 06:52:42.433727 29039 tablet_bootstrap.cc:492] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3: Bootstrap starting.
I20260621 06:52:42.434595 29037 tablet_bootstrap.cc:654] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:42.434532 29040 raft_consensus.cc:348] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.434917 29040 raft_consensus.cc:374] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:42.435017 29040 raft_consensus.cc:729] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f284c245134cc3b89c862385b1e222, State: Initialized, Role: FOLLOWER
I20260621 06:52:42.435251 29040 consensus_queue.cc:260] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.435437 29039 tablet_bootstrap.cc:654] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:42.435772 29040 ts_tablet_manager.cc:1428] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 06:52:42.444936 29039 tablet_bootstrap.cc:492] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3: No bootstrap required, opened a new log
I20260621 06:52:42.445201 29039 ts_tablet_manager.cc:1397] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260621 06:52:42.445211 29037 tablet_bootstrap.cc:492] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe: No bootstrap required, opened a new log
I20260621 06:52:42.445364 29037 ts_tablet_manager.cc:1397] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260621 06:52:42.446063 29037 raft_consensus.cc:348] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.446084 29039 raft_consensus.cc:348] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.446249 29037 raft_consensus.cc:374] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:42.446337 29037 raft_consensus.cc:729] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ad1765a77e44678de168473f3ccffe, State: Initialized, Role: FOLLOWER
I20260621 06:52:42.446355 29039 raft_consensus.cc:374] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:42.446439 29039 raft_consensus.cc:729] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da34b0cd81c46138c244cffa93469d3, State: Initialized, Role: FOLLOWER
I20260621 06:52:42.446583 29037 consensus_queue.cc:260] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.446722 29039 consensus_queue.cc:260] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.447224 29037 ts_tablet_manager.cc:1428] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 06:52:42.447849 29039 ts_tablet_manager.cc:1428] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 06:52:42.710151 29043 raft_consensus.cc:482] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:52:42.710438 29043 raft_consensus.cc:504] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.711232 29043 leader_election.cc:288] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193:38051), f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587)
I20260621 06:52:42.711853 28824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38263fae16e1404f85b68adf9b37206f" candidate_uuid: "e3ad1765a77e44678de168473f3ccffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f284c245134cc3b89c862385b1e222" is_pre_election: true
I20260621 06:52:42.711925 28687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38263fae16e1404f85b68adf9b37206f" candidate_uuid: "e3ad1765a77e44678de168473f3ccffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da34b0cd81c46138c244cffa93469d3" is_pre_election: true
I20260621 06:52:42.712131 28824 raft_consensus.cc:2454] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3ad1765a77e44678de168473f3ccffe in term 0.
I20260621 06:52:42.712214 28687 raft_consensus.cc:2454] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3ad1765a77e44678de168473f3ccffe in term 0.
I20260621 06:52:42.712561 28895 leader_election.cc:302] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3ad1765a77e44678de168473f3ccffe, f7f284c245134cc3b89c862385b1e222; no voters:
I20260621 06:52:42.712868 29043 raft_consensus.cc:2781] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:52:42.712980 29043 raft_consensus.cc:482] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:52:42.713090 29043 raft_consensus.cc:3037] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:42.716075 29043 raft_consensus.cc:504] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.716744 29043 leader_election.cc:288] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [CANDIDATE]: Term 1 election: Requested vote from peers 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193:38051), f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587)
I20260621 06:52:42.717324 28687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38263fae16e1404f85b68adf9b37206f" candidate_uuid: "e3ad1765a77e44678de168473f3ccffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da34b0cd81c46138c244cffa93469d3"
I20260621 06:52:42.717582 28687 raft_consensus.cc:3037] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:42.718291 28824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38263fae16e1404f85b68adf9b37206f" candidate_uuid: "e3ad1765a77e44678de168473f3ccffe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f284c245134cc3b89c862385b1e222"
I20260621 06:52:42.718502 28824 raft_consensus.cc:3037] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:42.719944 28687 raft_consensus.cc:2454] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3ad1765a77e44678de168473f3ccffe in term 1.
I20260621 06:52:42.720429 28897 leader_election.cc:302] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da34b0cd81c46138c244cffa93469d3, e3ad1765a77e44678de168473f3ccffe; no voters:
I20260621 06:52:42.720687 29043 raft_consensus.cc:2781] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:42.721171 29043 raft_consensus.cc:686] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 1 LEADER]: Becoming Leader. State: Replica: e3ad1765a77e44678de168473f3ccffe, State: Running, Role: LEADER
I20260621 06:52:42.721342 28824 raft_consensus.cc:2454] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3ad1765a77e44678de168473f3ccffe in term 1.
I20260621 06:52:42.721586 29043 consensus_queue.cc:237] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.730293 28527 catalog_manager.cc:5495] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe reported cstate change: term changed from 0 to 1, leader changed from <none> to e3ad1765a77e44678de168473f3ccffe (127.20.120.195). New cstate: current_term: 1 leader_uuid: "e3ad1765a77e44678de168473f3ccffe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } health_report { overall_health: HEALTHY } } }
I20260621 06:52:42.806329 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:42.799649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.20.120.195:37475)
0621 06:52:42.800390 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:42.800395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:42.800411 (+ 16us) server_negotiation.cc:407] Connection header received
0621 06:52:42.800508 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:42.800511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:42.800606 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:42.800745 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:52:42.801166 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.801864 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.803888 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:42.804359 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:42.804463 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:42.805001 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:52:42.805632 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:52:42.805736 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:42.805890 (+ 154us) server_negotiation.cc:299] Negotiation successful
0621 06:52:42.806003 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":535,"threads_started":1}
I20260621 06:52:42.809818 28824 raft_consensus.cc:1261] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Refusing update from remote peer e3ad1765a77e44678de168473f3ccffe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:42.809998 28687 raft_consensus.cc:1261] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Refusing update from remote peer e3ad1765a77e44678de168473f3ccffe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:42.810581 29043 consensus_queue.cc:1035] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:42.810917 29113 consensus_queue.cc:1035] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:42.852280 28528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:42.852569 28528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:42.856251 28528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb: Sending DeleteTablet for 3 replicas of tablet 38263fae16e1404f85b68adf9b37206f
I20260621 06:52:42.857318 28664 tablet_service.cc:1514] Processing DeleteTablet for tablet 38263fae16e1404f85b68adf9b37206f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:42 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:59474
I20260621 06:52:42.857520 28804 tablet_service.cc:1514] Processing DeleteTablet for tablet 38263fae16e1404f85b68adf9b37206f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:42 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:33204
I20260621 06:52:42.858153 28943 tablet_service.cc:1514] Processing DeleteTablet for tablet 38263fae16e1404f85b68adf9b37206f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:42 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:32936
I20260621 06:52:42.864814 28528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:52:42.865551 28528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:52:42.870067 29126 tablet_replica.cc:330] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3: stopping tablet replica
I20260621 06:52:42.870479 29126 raft_consensus.cc:2229] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:42.870898 29126 raft_consensus.cc:2258] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:42.872180 29126 ts_tablet_manager.cc:1905] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:42.879817 29126 ts_tablet_manager.cc:1918] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:42.880074 29126 log.cc:1196] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/38263fae16e1404f85b68adf9b37206f
I20260621 06:52:42.880877 29126 ts_tablet_manager.cc:1939] T 38263fae16e1404f85b68adf9b37206f P 9da34b0cd81c46138c244cffa93469d3: Deleting consensus metadata
I20260621 06:52:42.886123 28515 catalog_manager.cc:4841] TS 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193:38051): tablet 38263fae16e1404f85b68adf9b37206f (table test-table [id=8ccbd0a5e476426b8fc29cd46810560c]) successfully deleted
I20260621 06:52:42.889605 29128 tablet_replica.cc:330] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe: stopping tablet replica
I20260621 06:52:42.889972 29128 raft_consensus.cc:2229] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:52:42.890892 29127 tablet_replica.cc:330] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222: stopping tablet replica
I20260621 06:52:42.891181 29128 raft_consensus.cc:2258] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:42.891490 29127 raft_consensus.cc:2229] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:42.891796 29127 raft_consensus.cc:2258] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:42.893472 29128 ts_tablet_manager.cc:1905] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:42.896136 28803 tablet_service.cc:1467] Processing CreateTablet for tablet 60dd7929058c4521a73fa38246711c84 (DEFAULT_TABLE table=test-table [id=e362f36d6a694e49ab409ef21bf291a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:42.896591 28803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60dd7929058c4521a73fa38246711c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:42.896852 29127 ts_tablet_manager.cc:1905] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:42.902987 28664 tablet_service.cc:1467] Processing CreateTablet for tablet 60dd7929058c4521a73fa38246711c84 (DEFAULT_TABLE table=test-table [id=e362f36d6a694e49ab409ef21bf291a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:42.903466 28664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60dd7929058c4521a73fa38246711c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:42.906736 28942 tablet_service.cc:1467] Processing CreateTablet for tablet 60dd7929058c4521a73fa38246711c84 (DEFAULT_TABLE table=test-table [id=e362f36d6a694e49ab409ef21bf291a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:52:42.907310 28942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60dd7929058c4521a73fa38246711c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:42.909018 29127 ts_tablet_manager.cc:1918] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:42.909168 29127 log.cc:1196] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/38263fae16e1404f85b68adf9b37206f
I20260621 06:52:42.909623 29127 ts_tablet_manager.cc:1939] T 38263fae16e1404f85b68adf9b37206f P f7f284c245134cc3b89c862385b1e222: Deleting consensus metadata
I20260621 06:52:42.911046 28513 catalog_manager.cc:4841] TS f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587): tablet 38263fae16e1404f85b68adf9b37206f (table test-table [id=8ccbd0a5e476426b8fc29cd46810560c]) successfully deleted
I20260621 06:52:42.913095 29040 tablet_bootstrap.cc:492] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222: Bootstrap starting.
I20260621 06:52:42.914085 29039 tablet_bootstrap.cc:492] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3: Bootstrap starting.
I20260621 06:52:42.915004 29040 tablet_bootstrap.cc:654] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:42.916015 29039 tablet_bootstrap.cc:654] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:42.927052 29037 tablet_bootstrap.cc:492] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe: Bootstrap starting.
I20260621 06:52:42.929152 29037 tablet_bootstrap.cc:654] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:42.937423 29128 ts_tablet_manager.cc:1918] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:52:42.938416 29128 log.cc:1196] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/38263fae16e1404f85b68adf9b37206f
I20260621 06:52:42.939296 29128 ts_tablet_manager.cc:1939] T 38263fae16e1404f85b68adf9b37206f P e3ad1765a77e44678de168473f3ccffe: Deleting consensus metadata
I20260621 06:52:42.943426 28515 catalog_manager.cc:4841] TS e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475): tablet 38263fae16e1404f85b68adf9b37206f (table test-table [id=8ccbd0a5e476426b8fc29cd46810560c]) successfully deleted
I20260621 06:52:42.946696 29040 tablet_bootstrap.cc:492] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222: No bootstrap required, opened a new log
I20260621 06:52:42.946904 29040 ts_tablet_manager.cc:1397] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.009s
I20260621 06:52:42.947844 29040 raft_consensus.cc:348] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.948040 29037 tablet_bootstrap.cc:492] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe: No bootstrap required, opened a new log
I20260621 06:52:42.948117 29040 raft_consensus.cc:374] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:42.948197 29040 raft_consensus.cc:729] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f284c245134cc3b89c862385b1e222, State: Initialized, Role: FOLLOWER
I20260621 06:52:42.948407 29037 ts_tablet_manager.cc:1397] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.009s
I20260621 06:52:42.948499 29040 consensus_queue.cc:260] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.949398 29037 raft_consensus.cc:348] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.949762 29037 raft_consensus.cc:374] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:42.950233 29037 raft_consensus.cc:729] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ad1765a77e44678de168473f3ccffe, State: Initialized, Role: FOLLOWER
I20260621 06:52:42.950685 29037 consensus_queue.cc:260] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.952907 29040 ts_tablet_manager.cc:1428] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260621 06:52:42.953792 29037 ts_tablet_manager.cc:1428] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260621 06:52:42.957055 29039 tablet_bootstrap.cc:492] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3: No bootstrap required, opened a new log
I20260621 06:52:42.957311 29039 ts_tablet_manager.cc:1397] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.009s
I20260621 06:52:42.958178 29039 raft_consensus.cc:348] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.958456 29039 raft_consensus.cc:374] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:42.958575 29039 raft_consensus.cc:729] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da34b0cd81c46138c244cffa93469d3, State: Initialized, Role: FOLLOWER
I20260621 06:52:42.958868 29039 consensus_queue.cc:260] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.959518 29039 ts_tablet_manager.cc:1428] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 06:52:42.993839 29045 raft_consensus.cc:482] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:52:42.994100 29045 raft_consensus.cc:504] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:42.994903 29045 leader_election.cc:288] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587), e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475)
I20260621 06:52:42.995471 28824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60dd7929058c4521a73fa38246711c84" candidate_uuid: "9da34b0cd81c46138c244cffa93469d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f284c245134cc3b89c862385b1e222" is_pre_election: true
I20260621 06:52:42.995478 28963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60dd7929058c4521a73fa38246711c84" candidate_uuid: "9da34b0cd81c46138c244cffa93469d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ad1765a77e44678de168473f3ccffe" is_pre_election: true
I20260621 06:52:42.995762 28824 raft_consensus.cc:2454] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da34b0cd81c46138c244cffa93469d3 in term 0.
I20260621 06:52:42.995852 28963 raft_consensus.cc:2454] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da34b0cd81c46138c244cffa93469d3 in term 0.
I20260621 06:52:42.996290 28618 leader_election.cc:302] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da34b0cd81c46138c244cffa93469d3, e3ad1765a77e44678de168473f3ccffe; no voters:
I20260621 06:52:42.996632 29045 raft_consensus.cc:2781] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:52:42.996773 29045 raft_consensus.cc:482] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:52:42.996870 29045 raft_consensus.cc:3037] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:42.999233 29045 raft_consensus.cc:504] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:43.000237 28963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60dd7929058c4521a73fa38246711c84" candidate_uuid: "9da34b0cd81c46138c244cffa93469d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ad1765a77e44678de168473f3ccffe"
I20260621 06:52:43.000232 28824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60dd7929058c4521a73fa38246711c84" candidate_uuid: "9da34b0cd81c46138c244cffa93469d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f284c245134cc3b89c862385b1e222"
I20260621 06:52:43.000435 28963 raft_consensus.cc:3037] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:43.000435 28824 raft_consensus.cc:3037] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:43.003383 28824 raft_consensus.cc:2454] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da34b0cd81c46138c244cffa93469d3 in term 1.
I20260621 06:52:43.003400 28963 raft_consensus.cc:2454] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da34b0cd81c46138c244cffa93469d3 in term 1.
I20260621 06:52:43.003782 28618 leader_election.cc:302] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da34b0cd81c46138c244cffa93469d3, e3ad1765a77e44678de168473f3ccffe; no voters:
I20260621 06:52:43.004292 29045 leader_election.cc:288] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [CANDIDATE]: Term 1 election: Requested vote from peers f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587), e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475)
I20260621 06:52:43.004428 29132 raft_consensus.cc:2781] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:43.004637 29132 raft_consensus.cc:686] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 1 LEADER]: Becoming Leader. State: Replica: 9da34b0cd81c46138c244cffa93469d3, State: Running, Role: LEADER
I20260621 06:52:43.004987 29132 consensus_queue.cc:237] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } }
I20260621 06:52:43.007934 28527 catalog_manager.cc:5495] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193). New cstate: current_term: 1 leader_uuid: "9da34b0cd81c46138c244cffa93469d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da34b0cd81c46138c244cffa93469d3" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 } health_report { overall_health: UNKNOWN } } }
I20260621 06:52:43.045665 29134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:43.030317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.20.120.193:38051)
0621 06:52:43.038341 (+ 8024us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:43.038348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:43.038378 (+ 30us) server_negotiation.cc:407] Connection header received
0621 06:52:43.038462 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:43.038467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:43.038558 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:43.038690 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:52:43.039272 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:43.040536 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:43.042698 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:43.043452 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:43.044156 (+ 704us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:43.044319 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:52:43.044492 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:52:43.045153 (+ 661us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:43.045203 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 06:52:43.045280 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7877,"thread_start_us":7726,"threads_started":1}
I20260621 06:52:43.057703 28824 raft_consensus.cc:1261] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Refusing update from remote peer 9da34b0cd81c46138c244cffa93469d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:43.057696 28963 raft_consensus.cc:1261] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Refusing update from remote peer 9da34b0cd81c46138c244cffa93469d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:52:43.058247 29132 consensus_queue.cc:1035] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f284c245134cc3b89c862385b1e222" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:43.058509 29045 consensus_queue.cc:1035] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ad1765a77e44678de168473f3ccffe" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:52:43.135175 28528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:43.135457 28528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:52:43.138412 28528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 569e664927304215931d87758f4910eb: Sending DeleteTablet for 3 replicas of tablet 60dd7929058c4521a73fa38246711c84
I20260621 06:52:43.139158 28665 tablet_service.cc:1514] Processing DeleteTablet for tablet 60dd7929058c4521a73fa38246711c84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:43 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:59474
I20260621 06:52:43.139319 28803 tablet_service.cc:1514] Processing DeleteTablet for tablet 60dd7929058c4521a73fa38246711c84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:43 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:33204
I20260621 06:52:43.139596 29127 tablet_replica.cc:330] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222: stopping tablet replica
I20260621 06:52:43.139601 29126 tablet_replica.cc:330] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3: stopping tablet replica
I20260621 06:52:43.139617 28942 tablet_service.cc:1514] Processing DeleteTablet for tablet 60dd7929058c4521a73fa38246711c84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:52:43 UTC) from {username='oryx', principal='oryx/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:32936
I20260621 06:52:43.139772 29127 raft_consensus.cc:2229] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:43.139799 29126 raft_consensus.cc:2229] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:52:43.139957 29127 raft_consensus.cc:2258] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:43.140034 29126 raft_consensus.cc:2258] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:43.140234 29128 tablet_replica.cc:330] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe: stopping tablet replica
I20260621 06:52:43.140432 29128 raft_consensus.cc:2229] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:52:43.140610 29128 raft_consensus.cc:2258] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:52:43.141328 29127 ts_tablet_manager.cc:1905] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:43.141554 29126 ts_tablet_manager.cc:1905] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:43.142531 29128 ts_tablet_manager.cc:1905] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:52:43.147742 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 28599
I20260621 06:52:43.148480 29127 ts_tablet_manager.cc:1918] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:52:43.148648 29127 log.cc:1196] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/60dd7929058c4521a73fa38246711c84
I20260621 06:52:43.149102 29127 ts_tablet_manager.cc:1939] T 60dd7929058c4521a73fa38246711c84 P f7f284c245134cc3b89c862385b1e222: Deleting consensus metadata
I20260621 06:52:43.150281 28513 catalog_manager.cc:4841] TS f7f284c245134cc3b89c862385b1e222 (127.20.120.194:44587): tablet 60dd7929058c4521a73fa38246711c84 (table test-table [id=e362f36d6a694e49ab409ef21bf291a7]) successfully deleted
I20260621 06:52:43.150346 29128 ts_tablet_manager.cc:1918] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:52:43.150600 29128 log.cc:1196] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/60dd7929058c4521a73fa38246711c84
I20260621 06:52:43.151036 29128 ts_tablet_manager.cc:1939] T 60dd7929058c4521a73fa38246711c84 P e3ad1765a77e44678de168473f3ccffe: Deleting consensus metadata
I20260621 06:52:43.152235 28515 catalog_manager.cc:4841] TS e3ad1765a77e44678de168473f3ccffe (127.20.120.195:37475): tablet 60dd7929058c4521a73fa38246711c84 (table test-table [id=e362f36d6a694e49ab409ef21bf291a7]) successfully deleted
I20260621 06:52:43.153862 29126 ts_tablet_manager.cc:1918] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 06:52:43.154044 29126 log.cc:1196] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/60dd7929058c4521a73fa38246711c84
I20260621 06:52:43.154472 29126 ts_tablet_manager.cc:1939] T 60dd7929058c4521a73fa38246711c84 P 9da34b0cd81c46138c244cffa93469d3: Deleting consensus metadata
I20260621 06:52:43.155547 28515 catalog_manager.cc:4841] TS 9da34b0cd81c46138c244cffa93469d3 (127.20.120.193:38051): tablet 60dd7929058c4521a73fa38246711c84 (table test-table [id=e362f36d6a694e49ab409ef21bf291a7]) successfully deleted
Jun 21 06:52:43 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.120.193@KRBTEST.COM
I20260621 06:52:43.170863 29134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:43.149823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.20.120.193:38051)
0621 06:52:43.150147 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:43.150156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:43.155730 (+ 5574us) server_negotiation.cc:407] Connection header received
0621 06:52:43.156009 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:43.156014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:43.156081 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:43.156173 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:43.157624 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:43.158409 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:43.160219 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:43.160734 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:43.160908 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:43.165122 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:43.165143 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:43.165158 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:43.165196 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:43.168139 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:43.168787 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:43.168798 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:43.168802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:43.168956 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:43.169354 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:43.169358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:43.169360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:43.169671 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:43.170052 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:43.170308 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 06:52:43.170533 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260621 06:52:43.171543 28730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:43.314165 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 28599
I20260621 06:52:43.340710 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 28739
Jun 21 06:52:43 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.120.194@KRBTEST.COM
I20260621 06:52:43.357801 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:43.341508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.20.120.194:44587)
0621 06:52:43.342112 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:43.342120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:43.342139 (+ 19us) server_negotiation.cc:407] Connection header received
0621 06:52:43.342207 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:43.342212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:43.342290 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:43.342407 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:43.343687 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:43.344748 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:43.345991 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:43.346639 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:43.346761 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:43.351000 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:43.351018 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:43.351037 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:43.351080 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:43.354832 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:43.355621 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:43.355631 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:43.355634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:43.355781 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:43.356128 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:43.356135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:43.356140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:43.356646 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:43.356978 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:43.357241 (+ 263us) server_negotiation.cc:299] Negotiation successful
0621 06:52:43.357450 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":270,"threads_started":1}
I20260621 06:52:43.358675 28869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:43.481997 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 28739
I20260621 06:52:43.504170 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 28878
Jun 21 06:52:43 dist-test-slave-gmf2 krb5kdc[28483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.120.195@KRBTEST.COM
I20260621 06:52:43.520287 29146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:43.504901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.20.120.195:37475)
0621 06:52:43.505414 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:43.505418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:43.505433 (+ 15us) server_negotiation.cc:407] Connection header received
0621 06:52:43.505494 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:43.505497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:43.505553 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:43.505673 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:43.507073 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:43.507783 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:43.509035 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:43.509450 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:43.509597 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:43.514391 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:43.514406 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:43.514423 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:43.514555 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:43.517410 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:43.518199 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:43.518207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:43.518209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:43.518345 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:43.518729 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:43.518733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:43.518735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:43.519085 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:43.519399 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:43.519720 (+ 321us) server_negotiation.cc:299] Negotiation successful
0621 06:52:43.519942 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":270,"threads_started":1}
I20260621 06:52:43.520987 29008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:43.642036 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 28878
I20260621 06:52:43.664117 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 28496
I20260621 06:52:43.665295 28568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:43.777870 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 28496
2026-06-21T06:52:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5617 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:52:43 dist-test-slave-gmf2 krb5kdc[29151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:52:43 dist-test-slave-gmf2 krb5kdc[29151](info): set up 2 sockets
Jun 21 06:52:43 dist-test-slave-gmf2 krb5kdc[29151](info): commencing operation
krb5kdc: starting...
W20260621 06:52:44.862785 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:52:45 dist-test-slave-gmf2 krb5kdc[29151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:52:45.087185 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:46617
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:33223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:46617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:45.410140 29164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:45.410499 29164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:45.410660 29164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:45.417855 29164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:45.418026 29164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:45.418128 29164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:45.418229 29164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:45.418315 29164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:45.428236 29164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:33223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:46617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:46617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:45.429641 29164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:45.431038 29164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:45.446187 29170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:45.446347 29169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:45.446187 29172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:45.447130 29164 server_base.cc:1034] running on GCE node
I20260621 06:52:45.447973 29164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:45.449457 29164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:45.450692 29164 hybrid_clock.cc:648] HybridClock initialized: now 1782024765450642 us; error 46 us; skew 500 ppm
I20260621 06:52:45.451058 29164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-21T06:52:45Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1736 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:52:45 dist-test-slave-gmf2 krb5kdc[29178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:52:45 dist-test-slave-gmf2 krb5kdc[29178](info): set up 2 sockets
Jun 21 06:52:45 dist-test-slave-gmf2 krb5kdc[29178](info): commencing operation
krb5kdc: starting...
W20260621 06:52:46.607194 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:52:46 dist-test-slave-gmf2 krb5kdc[29178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:52:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:52:46.826218 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44571
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:44571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:47.149789 29191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:47.150163 29191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:47.150327 29191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:47.157611 29191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:47.157716 29191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:47.157828 29191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:47.157960 29191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:47.158047 29191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:47.167721 29191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:44571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:47.169031 29191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:47.170367 29191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:47.182096 29199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:47.182092 29196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:47.182116 29197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:47.182844 29191 server_base.cc:1034] running on GCE node
I20260621 06:52:47.183661 29191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:47.185050 29191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:47.186276 29191 hybrid_clock.cc:648] HybridClock initialized: now 1782024767186228 us; error 56 us; skew 500 ppm
I20260621 06:52:47.186622 29191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:47 dist-test-slave-gmf2 krb5kdc[29178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024767, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:47.189625 29191 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:52:47.191075 29191 webserver.cc:460] Webserver started at http://127.20.120.254:35025/ using document root <none> and password file <none>
I20260621 06:52:47.191720 29191 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:47.191814 29191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:47.192091 29191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:47.195014 29191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "84d56311736344cdad3a9f95d7218ab1"
format_stamp: "Formatted at 2026-06-21 06:52:47 on dist-test-slave-gmf2"
server_key: "7e46a5e2c405c90ba388e4aef744ff59"
server_key_iv: "6f09ca6feef9e7b2c762e9a3cddc514f"
server_key_version: "encryptionkey@0"
I20260621 06:52:47.195791 29191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "84d56311736344cdad3a9f95d7218ab1"
format_stamp: "Formatted at 2026-06-21 06:52:47 on dist-test-slave-gmf2"
server_key: "7e46a5e2c405c90ba388e4aef744ff59"
server_key_iv: "6f09ca6feef9e7b2c762e9a3cddc514f"
server_key_version: "encryptionkey@0"
I20260621 06:52:47.200925 29191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 06:52:47.204314 29206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:47.205731 29191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:52:47.206008 29191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "84d56311736344cdad3a9f95d7218ab1"
format_stamp: "Formatted at 2026-06-21 06:52:47 on dist-test-slave-gmf2"
server_key: "7e46a5e2c405c90ba388e4aef744ff59"
server_key_iv: "6f09ca6feef9e7b2c762e9a3cddc514f"
server_key_version: "encryptionkey@0"
I20260621 06:52:47.206193 29191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:47.230158 29191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:47.234158 29191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:47.234447 29191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:47.251888 29191 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:44571
I20260621 06:52:47.251982 29257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:44571 every 8 connection(s)
I20260621 06:52:47.253266 29191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:47.257030 29258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:52:47.257189 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 29191
I20260621 06:52:47.257388 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:52:47.258121 20963 external_mini_cluster.cc:1428] Setting key 546c8fc8ee2fe32189a2ce84dd6ed573
I20260621 06:52:47.271240 29258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1: Bootstrap starting.
Jun 21 06:52:47 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:47.274183 29258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1: Neither blocks nor log segments found. Creating new log.
I20260621 06:52:47.275440 29258 log.cc:824] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:47.278719 29258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1: No bootstrap required, opened a new log
I20260621 06:52:47.282850 29261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:47.260015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.20.120.254:44571)
0621 06:52:47.260772 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:47.260786 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:47.260834 (+ 48us) server_negotiation.cc:407] Connection header received
0621 06:52:47.263784 (+ 2950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:47.263812 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:47.264319 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:47.264756 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:47.266208 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:47.268196 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:47.269594 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:47.270663 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:47.270836 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:47.274733 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:47.274744 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:47.274760 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:47.274804 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:47.278561 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:47.279247 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:47.279256 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:47.279270 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:47.279482 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:47.279822 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:47.279827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:47.279830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:47.280478 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:47.280773 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:47.281069 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 06:52:47.281502 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":300,"threads_started":1}
I20260621 06:52:47.288097 29258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d56311736344cdad3a9f95d7218ab1" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44571 } }
I20260621 06:52:47.288400 29258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:52:47.288467 29258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d56311736344cdad3a9f95d7218ab1, State: Initialized, Role: FOLLOWER
I20260621 06:52:47.288980 29258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d56311736344cdad3a9f95d7218ab1" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44571 } }
I20260621 06:52:47.289170 29258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:47.289259 29258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:47.289366 29258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:52:47.291404 29258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d56311736344cdad3a9f95d7218ab1" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44571 } }
I20260621 06:52:47.291811 29258 leader_election.cc:302] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84d56311736344cdad3a9f95d7218ab1; no voters:
I20260621 06:52:47.292320 29258 leader_election.cc:288] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:52:47.292469 29263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:52:47.293151 29263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 1 LEADER]: Becoming Leader. State: Replica: 84d56311736344cdad3a9f95d7218ab1, State: Running, Role: LEADER
I20260621 06:52:47.293653 29263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d56311736344cdad3a9f95d7218ab1" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44571 } }
I20260621 06:52:47.294348 29258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:47.296491 29265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84d56311736344cdad3a9f95d7218ab1. Latest consensus state: current_term: 1 leader_uuid: "84d56311736344cdad3a9f95d7218ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d56311736344cdad3a9f95d7218ab1" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44571 } } }
I20260621 06:52:47.296876 29265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:47.296726 29264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84d56311736344cdad3a9f95d7218ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d56311736344cdad3a9f95d7218ab1" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44571 } } }
I20260621 06:52:47.296974 29264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:47.298557 29271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:47.303567 29271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:47.312356 29271 catalog_manager.cc:1347] Generated new cluster ID: 9e63fe0bec334b3e859b9fc195d3fe28
I20260621 06:52:47.312458 29271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:47.344996 29271 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:52:47.346438 29271 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:47.355842 29271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1: Generated new TSK 0
I20260621 06:52:47.356731 29271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:52:47.414032 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44571
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:47.740235 29284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:47.740566 29284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:47.740741 29284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:47.747965 29284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:47.748116 29284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:47.748296 29284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:52:47.758087 29284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:47.759434 29284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:47.760756 29284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:47.773710 29289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:47.773648 29292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:47.773689 29290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:47.774631 29284 server_base.cc:1034] running on GCE node
I20260621 06:52:47.775210 29284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:47.776154 29284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:47.777386 29284 hybrid_clock.cc:648] HybridClock initialized: now 1782024767777318 us; error 59 us; skew 500 ppm
I20260621 06:52:47.777707 29284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:47 dist-test-slave-gmf2 krb5kdc[29178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024767, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:47.784158 29284 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:52:47.785593 29284 webserver.cc:460] Webserver started at http://127.20.120.193:41135/ using document root <none> and password file <none>
I20260621 06:52:47.786345 29284 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:47.786486 29284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:47.786803 29284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:47.789664 29284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "e10de9ae7e374ef9afd34689ffef9cfa"
format_stamp: "Formatted at 2026-06-21 06:52:47 on dist-test-slave-gmf2"
server_key: "1de8ae3916b084e82d4eab97ca04e9e7"
server_key_iv: "7b66bff1141abe6049ce74719e59d2e6"
server_key_version: "encryptionkey@0"
I20260621 06:52:47.790496 29284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "e10de9ae7e374ef9afd34689ffef9cfa"
format_stamp: "Formatted at 2026-06-21 06:52:47 on dist-test-slave-gmf2"
server_key: "1de8ae3916b084e82d4eab97ca04e9e7"
server_key_iv: "7b66bff1141abe6049ce74719e59d2e6"
server_key_version: "encryptionkey@0"
I20260621 06:52:47.795598 29284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 06:52:47.799046 29299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:47.800402 29284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:52:47.800578 29284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "e10de9ae7e374ef9afd34689ffef9cfa"
format_stamp: "Formatted at 2026-06-21 06:52:47 on dist-test-slave-gmf2"
server_key: "1de8ae3916b084e82d4eab97ca04e9e7"
server_key_iv: "7b66bff1141abe6049ce74719e59d2e6"
server_key_version: "encryptionkey@0"
I20260621 06:52:47.800772 29284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:47.814126 29284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:47.818215 29284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:47.818490 29284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:47.819411 29284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:47.820679 29284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:47.820779 29284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:47.820912 29284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:47.821010 29284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260621 06:52:47.854007 29284 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:40755
I20260621 06:52:47.854033 29411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:40755 every 8 connection(s)
I20260621 06:52:47.855301 29284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:52:47.858060 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 29284
I20260621 06:52:47.858273 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:52:47.859004 20963 external_mini_cluster.cc:1428] Setting key 37c284133c9aaec2076481bde02ec3cd
Jun 21 06:52:47 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024767, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:47.877229 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:47.857856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:59705 (local address 127.20.120.254:44571)
0621 06:52:47.858356 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:47.858362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:47.859538 (+ 1176us) server_negotiation.cc:407] Connection header received
0621 06:52:47.861726 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:47.861730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:47.861790 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:47.861876 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:47.864035 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:47.864721 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:47.867391 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:47.867783 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:47.867884 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:47.871610 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:47.871614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:47.871618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:47.871649 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:47.874154 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:47.874891 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:47.874895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:47.874898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:47.875015 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:47.875422 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:47.875427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:47.875432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:47.875713 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:47.875881 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:47.876595 (+ 714us) server_negotiation.cc:299] Negotiation successful
0621 06:52:47.876812 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":214,"threads_started":1}
I20260621 06:52:47.878337 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:47.858449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:59705)
0621 06:52:47.859320 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:47.859377 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:47.861426 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:47.862171 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:47.862186 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:47.862964 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:47.863841 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:47.863870 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:47.864893 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:47.864901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:47.867130 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:47.867147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:47.868019 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:47.868028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:47.868372 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:47.869169 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:47.869191 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:47.871368 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:47.874293 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:47.874302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:47.874321 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:47.874757 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:47.875123 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:47.875129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:47.875132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:47.875309 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:47.875845 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:47.875854 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:47.876214 (+ 360us) client_negotiation.cc:769] Sending connection context
0621 06:52:47.876498 (+ 284us) client_negotiation.cc:240] Negotiation successful
0621 06:52:47.877062 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":325,"threads_started":1}
I20260621 06:52:47.879922 29412 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44571
I20260621 06:52:47.880270 29412 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:47.881263 29412 heartbeater.cc:511] Master 127.20.120.254:44571 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
I20260621 06:52:47.883518 29223 ts_manager.cc:194] Registered new tserver with Master: e10de9ae7e374ef9afd34689ffef9cfa (127.20.120.193:40755)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
I20260621 06:52:47.884923 29223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:59705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:52:47.908900 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44571
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:48.232007 29418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:48.232321 29418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:48.232474 29418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:48.239600 29418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:48.239706 29418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:48.239899 29418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:52:48.249639 29418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:48.250957 29418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:48.252334 29418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:48.266667 29423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:48.266767 29424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:48.267257 29426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:48.267793 29418 server_base.cc:1034] running on GCE node
I20260621 06:52:48.268507 29418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:48.269644 29418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:48.270838 29418 hybrid_clock.cc:648] HybridClock initialized: now 1782024768270764 us; error 75 us; skew 500 ppm
I20260621 06:52:48.271165 29418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:48 dist-test-slave-gmf2 krb5kdc[29178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024768, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:48.274272 29418 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:52:48.275627 29418 webserver.cc:460] Webserver started at http://127.20.120.194:36289/ using document root <none> and password file <none>
I20260621 06:52:48.276381 29418 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:48.276532 29418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:48.276844 29418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:48.279763 29418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "aa09d12b7f944ed5bd5a76332234d6b8"
format_stamp: "Formatted at 2026-06-21 06:52:48 on dist-test-slave-gmf2"
server_key: "2d67794c10bd0c03d04b4360a0645e47"
server_key_iv: "0d1358a262964cc552199fed2b38325d"
server_key_version: "encryptionkey@0"
I20260621 06:52:48.280542 29418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "aa09d12b7f944ed5bd5a76332234d6b8"
format_stamp: "Formatted at 2026-06-21 06:52:48 on dist-test-slave-gmf2"
server_key: "2d67794c10bd0c03d04b4360a0645e47"
server_key_iv: "0d1358a262964cc552199fed2b38325d"
server_key_version: "encryptionkey@0"
I20260621 06:52:48.285504 29418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 06:52:48.288959 29433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:48.290277 29418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:52:48.290449 29418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "aa09d12b7f944ed5bd5a76332234d6b8"
format_stamp: "Formatted at 2026-06-21 06:52:48 on dist-test-slave-gmf2"
server_key: "2d67794c10bd0c03d04b4360a0645e47"
server_key_iv: "0d1358a262964cc552199fed2b38325d"
server_key_version: "encryptionkey@0"
I20260621 06:52:48.290659 29418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:48.310065 29418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:48.314211 29418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:48.314523 29418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:48.315491 29418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:48.316874 29418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:48.316979 29418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:48.317142 29418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:48.317217 29418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:48.349524 29418 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:38167
I20260621 06:52:48.349562 29545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:38167 every 8 connection(s)
I20260621 06:52:48.350857 29418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:52:48.360639 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 29418
I20260621 06:52:48.360801 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:52:48.361297 20963 external_mini_cluster.cc:1428] Setting key 074d53663a972629fa61694a8a4e746d
Jun 21 06:52:48 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024768, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:48.370370 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:48.353478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:32863 (local address 127.20.120.254:44571)
0621 06:52:48.353655 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:48.353660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:48.354780 (+ 1120us) server_negotiation.cc:407] Connection header received
0621 06:52:48.356565 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:48.356570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:48.356653 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:48.356758 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:48.358225 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:48.358940 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:48.360593 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:48.361119 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:48.361229 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:48.364701 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:48.364706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:48.364709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:48.364739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:48.367216 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:48.367916 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:48.367921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:48.367926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:48.368060 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:48.368585 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:48.368593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:48.368598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:48.368953 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:48.369110 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:48.369974 (+ 864us) server_negotiation.cc:299] Negotiation successful
0621 06:52:48.370138 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:52:48.371876 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:48.353967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:32863)
0621 06:52:48.354627 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:48.354667 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:48.356313 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:48.356982 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:48.356989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:48.357497 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:48.358055 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:48.358090 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:48.359127 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:48.359131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:48.360438 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:48.360454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:48.361339 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:48.361347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:48.361625 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:48.362362 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:48.362387 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:48.364492 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:48.367357 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:48.367365 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:48.367379 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:48.367780 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:48.368169 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:48.368174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:48.368177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:48.368423 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:48.369093 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:48.369103 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:48.369526 (+ 423us) client_negotiation.cc:769] Sending connection context
0621 06:52:48.369827 (+ 301us) client_negotiation.cc:240] Negotiation successful
0621 06:52:48.370441 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":249,"threads_started":1}
I20260621 06:52:48.373284 29546 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44571
I20260621 06:52:48.373587 29546 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:48.374334 29546 heartbeater.cc:511] Master 127.20.120.254:44571 requested a full tablet report, sending...
I20260621 06:52:48.375710 29223 ts_manager.cc:194] Registered new tserver with Master: aa09d12b7f944ed5bd5a76332234d6b8 (127.20.120.194:38167)
I20260621 06:52:48.376487 29223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:32863
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:52:48.408485 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44571
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:48.723661 29551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:48.723920 29551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:48.724088 29551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:48.731127 29551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:48.731249 29551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:48.731415 29551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:52:48.740756 29551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:48.742033 29551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:48.743232 29551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:48.755759 29559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:48.755693 29556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:48.755790 29557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:48.756903 29551 server_base.cc:1034] running on GCE node
I20260621 06:52:48.757530 29551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:48.758620 29551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:48.759841 29551 hybrid_clock.cc:648] HybridClock initialized: now 1782024768759793 us; error 47 us; skew 500 ppm
I20260621 06:52:48.760172 29551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:48 dist-test-slave-gmf2 krb5kdc[29178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024768, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:48.763386 29551 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:52:48.764796 29551 webserver.cc:460] Webserver started at http://127.20.120.195:39307/ using document root <none> and password file <none>
I20260621 06:52:48.765466 29551 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:48.765604 29551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:48.765947 29551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:52:48.768951 29551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "0d2d6446bd2d4f009c4f14e964c0c08a"
format_stamp: "Formatted at 2026-06-21 06:52:48 on dist-test-slave-gmf2"
server_key: "7855cae638f4c5897a29a062f3dca71c"
server_key_iv: "cad6cfd860b176b53f52bb0eaef9a797"
server_key_version: "encryptionkey@0"
I20260621 06:52:48.769872 29551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "0d2d6446bd2d4f009c4f14e964c0c08a"
format_stamp: "Formatted at 2026-06-21 06:52:48 on dist-test-slave-gmf2"
server_key: "7855cae638f4c5897a29a062f3dca71c"
server_key_iv: "cad6cfd860b176b53f52bb0eaef9a797"
server_key_version: "encryptionkey@0"
I20260621 06:52:48.774966 29551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 06:52:48.778275 29566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:48.779455 29551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:52:48.779630 29551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "0d2d6446bd2d4f009c4f14e964c0c08a"
format_stamp: "Formatted at 2026-06-21 06:52:48 on dist-test-slave-gmf2"
server_key: "7855cae638f4c5897a29a062f3dca71c"
server_key_iv: "cad6cfd860b176b53f52bb0eaef9a797"
server_key_version: "encryptionkey@0"
I20260621 06:52:48.779814 29551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:48.793040 29551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:48.797322 29551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:48.797600 29551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:48.798579 29551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:48.799950 29551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:48.800052 29551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:48.800177 29551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:48.800256 29551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:48.831682 29551 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:33643
I20260621 06:52:48.831724 29678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:33643 every 8 connection(s)
I20260621 06:52:48.832927 29551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:52:48.839419 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 29551
I20260621 06:52:48.839613 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:52:48.840147 20963 external_mini_cluster.cc:1428] Setting key 527fe0cc12deefa350038a48d9f68d36
Jun 21 06:52:48 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024768, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:52:48.852372 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:48.835209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:45643 (local address 127.20.120.254:44571)
0621 06:52:48.835411 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:48.835417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:48.836597 (+ 1180us) server_negotiation.cc:407] Connection header received
0621 06:52:48.838236 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:48.838240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:48.838338 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:48.838456 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:48.840357 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:48.841188 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:48.843286 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:48.843671 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:48.843768 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:48.846911 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:48.846915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:48.846918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:48.846946 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:48.849499 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:48.850248 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:48.850252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:48.850255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:48.850412 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:48.850783 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:48.850786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:48.850788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:48.851051 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:48.851174 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:48.851790 (+ 616us) server_negotiation.cc:299] Negotiation successful
0621 06:52:48.851949 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:52:48.853449 29681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:48.835730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:45643)
0621 06:52:48.836425 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:48.836475 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:48.837996 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:48.838660 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:48.838667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:48.839342 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:48.840175 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:48.840201 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:48.841350 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:48.841355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:48.843145 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:48.843161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:48.843851 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:48.843855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:48.844043 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:48.844643 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:48.844663 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:48.846743 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:52:48.849628 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:48.849634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:48.849647 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:48.850111 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:48.850509 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:52:48.850512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:52:48.850515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:52:48.850681 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:52:48.851168 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:52:48.851173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:52:48.851494 (+ 321us) client_negotiation.cc:769] Sending connection context
0621 06:52:48.851707 (+ 213us) client_negotiation.cc:240] Negotiation successful
0621 06:52:48.852154 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":266,"threads_started":1}
I20260621 06:52:48.854990 29679 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44571
I20260621 06:52:48.855333 29679 heartbeater.cc:464] Registering TS with master...
I20260621 06:52:48.856160 29679 heartbeater.cc:511] Master 127.20.120.254:44571 requested a full tablet report, sending...
I20260621 06:52:48.857467 29223 ts_manager.cc:194] Registered new tserver with Master: 0d2d6446bd2d4f009c4f14e964c0c08a (127.20.120.195:33643)
I20260621 06:52:48.858328 29223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:45643
I20260621 06:52:48.868058 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.120.193@KRBTEST.COM; defaulting to no policy
I20260621 06:52:48.887694 29412 heartbeater.cc:503] Master 127.20.120.254:44571 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab.
Entry for principal oryx/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab.
WARNING: no policy specified for oryx/127.20.120.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab.
Entry for principal oryx/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab.
WARNING: no policy specified for oryx/127.20.120.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab.
Entry for principal oryx/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab.
I20260621 06:52:48.990387 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 29284
Jun 21 06:52:48 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:52:49.006487 29689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:48.991582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34226 (local address 127.20.120.193:40755)
0621 06:52:48.992224 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:48.992231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:48.992257 (+ 26us) server_negotiation.cc:407] Connection header received
0621 06:52:48.992315 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:48.992322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:48.992569 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:48.992700 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:48.994182 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:48.995298 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:48.996597 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:48.997140 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:48.997238 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:49.001366 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:49.001378 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:49.001392 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:49.001426 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:49.004112 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:49.004709 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:49.004716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:49.004718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:49.004830 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:49.005179 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:49.005182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:49.005184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:49.005523 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:49.005745 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:49.006030 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 06:52:49.006214 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":336,"threads_started":1}
I20260621 06:52:49.007746 29407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:49.089974 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 29284
I20260621 06:52:49.107208 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 29418
Jun 21 06:52:49 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:52:49.122490 29691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:49.108069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.20.120.194:38167)
0621 06:52:49.108630 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:49.108637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:49.108662 (+ 25us) server_negotiation.cc:407] Connection header received
0621 06:52:49.108727 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:49.108733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:49.108986 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:49.109120 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:49.110449 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:49.111534 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:49.112691 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:49.113204 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:49.113301 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:49.117295 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:49.117307 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:49.117322 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:49.117356 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:49.120150 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:49.120734 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:49.120741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:49.120743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:49.120855 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:49.121160 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:49.121163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:49.121165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:49.121492 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:49.121711 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:49.121997 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 06:52:49.122194 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":320,"threads_started":1}
I20260621 06:52:49.123828 29541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:49.204746 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 29418
I20260621 06:52:49.220541 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 29551
Jun 21 06:52:49 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:52:49.236519 29693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:49.221500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33972 (local address 127.20.120.195:33643)
0621 06:52:49.222216 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:49.222222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:49.222249 (+ 27us) server_negotiation.cc:407] Connection header received
0621 06:52:49.222334 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:49.222339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:49.222555 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:49.222673 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:49.223946 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:49.225093 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:49.226436 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:49.227003 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:49.227109 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:49.231365 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:49.231377 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:49.231391 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:49.231424 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:49.234236 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:49.234827 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:49.234851 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:49.234854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:49.234967 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:49.235266 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:49.235269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:49.235271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:49.235594 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:49.235824 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:49.236038 (+ 214us) server_negotiation.cc:299] Negotiation successful
0621 06:52:49.236221 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":320,"threads_started":1}
I20260621 06:52:49.237784 29674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:49.318543 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 29551
I20260621 06:52:49.333835 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 29191
I20260621 06:52:49.334923 29253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:52:49.435959 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 29191
I20260621 06:52:49.459400 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44571
--webserver_interface=127.20.120.254
--webserver_port=35025
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:44571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:52:49.774317 29696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:49.774569 29696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:49.774710 29696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:49.781498 29696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:52:49.781626 29696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:49.781670 29696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:49.781718 29696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:52:49.781801 29696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:52:49.791074 29696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:44571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=35025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:49.792321 29696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:49.793637 29696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:49.804911 29701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:49.804936 29704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:49.804877 29702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:49.806087 29696 server_base.cc:1034] running on GCE node
I20260621 06:52:49.806753 29696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:49.808068 29696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:49.809284 29696 hybrid_clock.cc:648] HybridClock initialized: now 1782024769809240 us; error 55 us; skew 500 ppm
I20260621 06:52:49.809623 29696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:49 dist-test-slave-gmf2 krb5kdc[29178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024769, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:49.812680 29696 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:52:49.814193 29696 webserver.cc:460] Webserver started at http://127.20.120.254:35025/ using document root <none> and password file <none>
I20260621 06:52:49.814854 29696 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:49.815006 29696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:49.820225 29696 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260621 06:52:49.822825 29711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:49.824414 29696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:52:49.824543 29696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "84d56311736344cdad3a9f95d7218ab1"
format_stamp: "Formatted at 2026-06-21 06:52:47 on dist-test-slave-gmf2"
server_key: "7e46a5e2c405c90ba388e4aef744ff59"
server_key_iv: "6f09ca6feef9e7b2c762e9a3cddc514f"
server_key_version: "encryptionkey@0"
I20260621 06:52:49.825438 29696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:49.845297 29696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:49.848942 29696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:49.849166 29696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:49.866061 29696 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:44571
I20260621 06:52:49.866065 29762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:44571 every 8 connection(s)
I20260621 06:52:49.867532 29696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:52:49.869396 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 29696
I20260621 06:52:49.870268 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:40755
--local_ip_for_outbound_sockets=127.20.120.193
--tserver_master_addrs=127.20.120.254:44571
--webserver_port=41135
--webserver_interface=127.20.120.193
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.120.193
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:52:49.874269 29763 sys_catalog.cc:263] Verifying existing consensus state
I20260621 06:52:49.878520 29763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1: Bootstrap starting.
I20260621 06:52:49.906140 29763 log.cc:824] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1: Log is configured to *not* fsync() on all Append() calls
I20260621 06:52:49.916698 29763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 06:52:49.917316 29763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1: Bootstrap complete.
I20260621 06:52:49.930917 29763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d56311736344cdad3a9f95d7218ab1" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44571 } }
I20260621 06:52:49.932065 29763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d56311736344cdad3a9f95d7218ab1, State: Initialized, Role: FOLLOWER
I20260621 06:52:49.932770 29763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d56311736344cdad3a9f95d7218ab1" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44571 } }
I20260621 06:52:49.933106 29763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:52:49.933265 29763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:52:49.933471 29763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 1 FOLLOWER]: Advancing to term 2
I20260621 06:52:49.939663 29763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d56311736344cdad3a9f95d7218ab1" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44571 } }
I20260621 06:52:49.940243 29763 leader_election.cc:302] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84d56311736344cdad3a9f95d7218ab1; no voters:
I20260621 06:52:49.940871 29763 leader_election.cc:288] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 06:52:49.941048 29767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 06:52:49.942404 29767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [term 2 LEADER]: Becoming Leader. State: Replica: 84d56311736344cdad3a9f95d7218ab1, State: Running, Role: LEADER
I20260621 06:52:49.942966 29763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:52:49.942970 29767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d56311736344cdad3a9f95d7218ab1" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44571 } }
I20260621 06:52:49.945307 29769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84d56311736344cdad3a9f95d7218ab1. Latest consensus state: current_term: 2 leader_uuid: "84d56311736344cdad3a9f95d7218ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d56311736344cdad3a9f95d7218ab1" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44571 } } }
I20260621 06:52:49.945662 29769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:49.946117 29768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "84d56311736344cdad3a9f95d7218ab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d56311736344cdad3a9f95d7218ab1" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44571 } } }
I20260621 06:52:49.946431 29768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1 [sys.catalog]: This master's current role is: LEADER
I20260621 06:52:49.955466 29774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:52:49.961138 29774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:52:49.962463 29774 catalog_manager.cc:1259] Loaded cluster ID: 9e63fe0bec334b3e859b9fc195d3fe28
I20260621 06:52:49.962607 29774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:52:49.966572 29774 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:52:49.967973 29774 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 84d56311736344cdad3a9f95d7218ab1: Loaded TSK: 0
I20260621 06:52:49.969528 29774 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 06:52:50.229241 29765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:50.229493 29765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:50.229645 29765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:50.236768 29765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:50.236871 29765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:50.237051 29765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:52:50.246806 29765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.193.keytab
--principal=oryx/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:40755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=41135
--tserver_master_addrs=127.20.120.254:44571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:50.248129 29765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:50.249732 29765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:50.263190 29790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:50.263675 29789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:50.264176 29792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:50.264202 29765 server_base.cc:1034] running on GCE node
I20260621 06:52:50.264981 29765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:50.266142 29765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:50.267408 29765 hybrid_clock.cc:648] HybridClock initialized: now 1782024770267355 us; error 58 us; skew 500 ppm
I20260621 06:52:50.267740 29765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024770, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:50.271476 29765 init.cc:377] Logged in from keytab as oryx/127.20.120.193@KRBTEST.COM (short username oryx)
I20260621 06:52:50.273075 29765 webserver.cc:460] Webserver started at http://127.20.120.193:41135/ using document root <none> and password file <none>
I20260621 06:52:50.273967 29765 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:50.274112 29765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:50.280877 29765 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260621 06:52:50.284189 29799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:50.285738 29765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 06:52:50.285899 29765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "e10de9ae7e374ef9afd34689ffef9cfa"
format_stamp: "Formatted at 2026-06-21 06:52:47 on dist-test-slave-gmf2"
server_key: "1de8ae3916b084e82d4eab97ca04e9e7"
server_key_iv: "7b66bff1141abe6049ce74719e59d2e6"
server_key_version: "encryptionkey@0"
I20260621 06:52:50.287189 29765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:50.311158 29765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:50.316252 29765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:50.316517 29765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:50.317570 29765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:50.319190 29765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:50.319304 29765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:50.319429 29765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:50.319511 29765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:50.351047 29765 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:40755
I20260621 06:52:50.351120 29911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:40755 every 8 connection(s)
I20260621 06:52:50.352633 29765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:52:50.355547 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 29765
I20260621 06:52:50.356309 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:38167
--local_ip_for_outbound_sockets=127.20.120.194
--tserver_master_addrs=127.20.120.254:44571
--webserver_port=36289
--webserver_interface=127.20.120.194
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.120.194
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:50.371699 29915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:50.355632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:47057)
0621 06:52:50.356780 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:50.356824 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:50.358723 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:50.360136 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:50.360147 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:50.360732 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:50.361267 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:50.361284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.362991 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.362998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:50.364715 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:50.364728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.366274 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.366279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:50.366479 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:50.367194 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:50.367234 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:50.370791 (+ 3557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":691,"threads_started":1}
W20260621 06:52:50.372438 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:50.355614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:47057 (local address 127.20.120.254:44571)
0621 06:52:50.356382 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:50.356398 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:50.356954 (+ 556us) server_negotiation.cc:407] Connection header received
0621 06:52:50.359266 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:50.359279 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:50.359623 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:50.359986 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:50.361463 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.362768 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.364860 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.366060 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.366267 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:50.371210 (+ 4943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:47057: BlockingRecv error: recv got EOF from 127.20.120.193:47057 (error 108)
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":247,"threads_started":1}
W20260621 06:52:50.372555 29912 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:44571 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.120.254:44571: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:50.381554 29915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:50.373147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:39993)
0621 06:52:50.373427 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:50.373448 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:50.373550 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:50.374005 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:50.374008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:50.374298 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:50.374436 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:50.374443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.376032 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.376036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:50.376849 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:50.376858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.377852 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.377856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:50.378030 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:50.378592 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:50.378608 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:50.381365 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164}
W20260621 06:52:50.381799 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:50.373422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:39993 (local address 127.20.120.254:44571)
0621 06:52:50.373629 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:50.373637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:50.373659 (+ 22us) server_negotiation.cc:407] Connection header received
0621 06:52:50.373717 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:50.373724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:50.373806 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:50.373940 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:50.374571 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.375893 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.376982 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.377723 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.377866 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:50.381598 (+ 3732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:39993: BlockingRecv error: recv got EOF from 127.20.120.193:39993 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:50.390009 29915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:50.382287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:55093)
0621 06:52:50.382537 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:50.382555 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:50.382643 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:50.382932 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:50.382935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:50.383261 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:50.383389 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:50.383397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.384695 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.384698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:50.385536 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:50.385545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.386434 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.386439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:50.386588 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:50.387155 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:50.387171 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:50.389745 (+ 2574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125}
W20260621 06:52:50.390262 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:50.382457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:55093 (local address 127.20.120.254:44571)
0621 06:52:50.382632 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:50.382638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:50.382655 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:52:50.382737 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:50.382742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:50.382819 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:50.382908 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:50.383516 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.384573 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.385653 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.386310 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.386428 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:50.390010 (+ 3582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:55093: BlockingRecv error: recv got EOF from 127.20.120.193:55093 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260621 06:52:50.390429 29912 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 06:52:50.686322 29916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:50.686579 29916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:50.686671 29916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:50.693789 29916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:50.693889 29916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:50.694064 29916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:52:50.704597 29916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.194.keytab
--principal=oryx/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:38167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=36289
--tserver_master_addrs=127.20.120.254:44571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:50.705870 29916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:50.707487 29916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:50.727828 29922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:50.728292 29921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:50.729030 29924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:50.729058 29916 server_base.cc:1034] running on GCE node
I20260621 06:52:50.729749 29916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:50.731353 29916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:50.732612 29916 hybrid_clock.cc:648] HybridClock initialized: now 1782024770732486 us; error 92 us; skew 500 ppm
I20260621 06:52:50.733093 29916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024770, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:50.737727 29916 init.cc:377] Logged in from keytab as oryx/127.20.120.194@KRBTEST.COM (short username oryx)
I20260621 06:52:50.739916 29916 webserver.cc:460] Webserver started at http://127.20.120.194:36289/ using document root <none> and password file <none>
I20260621 06:52:50.740772 29916 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:50.740940 29916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:50.748842 29916 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 06:52:50.752630 29931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:50.754489 29916 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260621 06:52:50.754676 29916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "aa09d12b7f944ed5bd5a76332234d6b8"
format_stamp: "Formatted at 2026-06-21 06:52:48 on dist-test-slave-gmf2"
server_key: "2d67794c10bd0c03d04b4360a0645e47"
server_key_iv: "0d1358a262964cc552199fed2b38325d"
server_key_version: "encryptionkey@0"
I20260621 06:52:50.756326 29916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:50.779325 29916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:50.783803 29916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:50.784030 29916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:50.784920 29916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:50.786311 29916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:50.786414 29916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:50.786510 29916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:50.786571 29916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:50.819950 29916 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:38167
I20260621 06:52:50.820003 30043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:38167 every 8 connection(s)
I20260621 06:52:50.821476 29916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:52:50.830869 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 29916
I20260621 06:52:50.831429 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:33643
--local_ip_for_outbound_sockets=127.20.120.195
--tserver_master_addrs=127.20.120.254:44571
--webserver_port=39307
--webserver_interface=127.20.120.195
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.120.195
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 06:52:50.838093 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:50.824079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:42919 (local address 127.20.120.254:44571)
0621 06:52:50.824269 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:50.824274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:50.825462 (+ 1188us) server_negotiation.cc:407] Connection header received
0621 06:52:50.827237 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:50.827241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:50.827337 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:50.827462 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:50.829011 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.829793 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.831398 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.831900 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.832051 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:50.837777 (+ 5726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:42919: BlockingRecv error: recv got EOF from 127.20.120.194:42919 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 06:52:50.838562 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:50.824548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:42919)
0621 06:52:50.825282 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:50.825324 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:50.826977 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:50.827701 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:50.827709 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:50.828314 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:50.828844 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:50.828859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.829979 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.829983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:50.831257 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:50.831269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.832160 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.832169 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:50.832390 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:50.833059 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:50.833086 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:50.837488 (+ 4402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":337,"threads_started":1}
W20260621 06:52:50.839188 30044 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:44571 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.120.254:44571: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:50.848608 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:50.839727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:52787)
0621 06:52:50.839949 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:50.839969 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:50.840073 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:50.840379 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:50.840386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:50.840819 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:50.841025 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:50.841039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.841986 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.841993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:50.843662 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:50.843677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.844417 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.844426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:50.844636 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:50.845232 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:50.845259 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:50.848405 (+ 3146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260621 06:52:50.848996 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:50.839857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:52787 (local address 127.20.120.254:44571)
0621 06:52:50.839986 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:50.839990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:50.840011 (+ 21us) server_negotiation.cc:407] Connection header received
0621 06:52:50.840186 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:50.840189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:50.840242 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:50.840320 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:50.841157 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.841825 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.843787 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.844260 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.844358 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:50.848704 (+ 4346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:52787: BlockingRecv error: recv got EOF from 127.20.120.194:52787 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:50 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:50.858901 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:50.849415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:43925)
0621 06:52:50.849614 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:50.849635 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:50.849745 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:50.850133 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:50.850140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:50.850589 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:50.850825 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:50.850837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.851776 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.851783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:50.853569 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:50.853587 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.854404 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.854411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:50.854657 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:50.855232 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:50.855258 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:50.858620 (+ 3362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260621 06:52:50.858932 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:50.849511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:43925 (local address 127.20.120.254:44571)
0621 06:52:50.849664 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:50.849668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:50.849684 (+ 16us) server_negotiation.cc:407] Connection header received
0621 06:52:50.849875 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:50.849879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:50.849988 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:50.850075 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:50.850968 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.851634 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.853705 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:50.854255 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:50.854342 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:50.858780 (+ 4438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:43925: BlockingRecv error: recv got EOF from 127.20.120.194:43925 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260621 06:52:50.859201 30044 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 06:52:51.153447 30047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:52:51.153774 30047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:52:51.153956 30047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:52:51.161278 30047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:52:51.161376 30047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:52:51.161511 30047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:52:51.171214 30047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/krb5kdc/oryx_127.20.120.195.keytab
--principal=oryx/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:33643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=39307
--tserver_master_addrs=127.20.120.254:44571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:52:51.172505 30047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:52:51.173759 30047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:52:51.186688 30052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:51.186655 30053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:52:51.186618 30055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:52:51.187471 30047 server_base.cc:1034] running on GCE node
I20260621 06:52:51.188084 30047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:52:51.189028 30047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:52:51.190281 30047 hybrid_clock.cc:648] HybridClock initialized: now 1782024771190205 us; error 40 us; skew 500 ppm
I20260621 06:52:51.190600 30047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:52:51 dist-test-slave-gmf2 krb5kdc[29178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024771, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:52:51.193787 30047 init.cc:377] Logged in from keytab as oryx/127.20.120.195@KRBTEST.COM (short username oryx)
I20260621 06:52:51.195386 30047 webserver.cc:460] Webserver started at http://127.20.120.195:39307/ using document root <none> and password file <none>
I20260621 06:52:51.196080 30047 fs_manager.cc:359] Metadata directory not provided
I20260621 06:52:51.196230 30047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:52:51.201695 30047 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:52:51.204319 30062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:51.205652 30047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:52:51.205816 30047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "0d2d6446bd2d4f009c4f14e964c0c08a"
format_stamp: "Formatted at 2026-06-21 06:52:48 on dist-test-slave-gmf2"
server_key: "7855cae638f4c5897a29a062f3dca71c"
server_key_iv: "cad6cfd860b176b53f52bb0eaef9a797"
server_key_version: "encryptionkey@0"
I20260621 06:52:51.206775 30047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:52:51.220644 30047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:52:51.224418 30047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:52:51.224692 30047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:52:51.225621 30047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:52:51.226961 30047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:52:51.227101 30047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:51.227243 30047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:52:51.227329 30047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:52:51.259941 30047 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:33643
I20260621 06:52:51.259991 30174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:33643 every 8 connection(s)
I20260621 06:52:51.261449 30047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:52:51.265848 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 30047
Jun 21 06:52:51 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:51 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 06:52:51.277174 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:51.263923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:36157 (local address 127.20.120.254:44571)
0621 06:52:51.264100 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:51.264104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:51.265177 (+ 1073us) server_negotiation.cc:407] Connection header received
0621 06:52:51.267598 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:51.267603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:51.267708 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:51.267813 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:51.269191 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.270012 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.271497 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.272026 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.272194 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:51.276885 (+ 4691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:36157: BlockingRecv error: recv got EOF from 127.20.120.195:36157 (error 108)
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":60}
I20260621 06:52:51.277801 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:51.264383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:36157)
0621 06:52:51.265032 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:51.265069 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:51.267266 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:51.268045 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:51.268053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:51.268502 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:51.269031 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:51.269046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.270157 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.270167 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:51.271364 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:51.271374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.272276 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.272283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:51.272603 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:51.273264 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:51.273285 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:51.276694 (+ 3409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":251,"threads_started":1}
W20260621 06:52:51.278507 30175 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:44571 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.120.254:44571: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
I20260621 06:52:51.281073 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:51.266909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.20.120.254:44571)
0621 06:52:51.267606 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:51.267614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:51.267631 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:52:51.267682 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:51.267689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:51.267743 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:51.267851 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:51.269231 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.270126 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.271386 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.271814 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.271952 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:51.274611 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:52:51.274619 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:52:51.274633 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:52:51.274690 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:52:51.277607 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:51.278415 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:51.278422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:51.278428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:51.278625 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:52:51.278989 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:52:51.278995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:52:51.278999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:52:51.279592 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:52:51.279833 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0621 06:52:51.280243 (+ 410us) server_negotiation.cc:299] Negotiation successful
0621 06:52:51.280647 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":585,"thread_start_us":453,"threads_started":1}
Jun 21 06:52:51 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:51 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:51.286839 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:51.279062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:57875)
0621 06:52:51.279224 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:51.279249 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:51.279320 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:51.279609 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:51.279612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:51.279991 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:51.280189 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:51.280200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.281534 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.281539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:51.282674 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:51.282685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.283538 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.283543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:51.283694 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:51.284208 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:51.284224 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:51.286672 (+ 2448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 06:52:51.287050 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:51.279135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:57875 (local address 127.20.120.254:44571)
0621 06:52:51.279285 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:51.279290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:51.279306 (+ 16us) server_negotiation.cc:407] Connection header received
0621 06:52:51.279411 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:51.279416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:51.279479 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:51.279577 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:51.280319 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.281409 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.282797 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.283418 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.283526 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:51.286815 (+ 3289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:57875: BlockingRecv error: recv got EOF from 127.20.120.195:57875 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 21 06:52:51 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:51 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:51.294219 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:51.287416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:43183)
0621 06:52:51.287567 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:51.287583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:51.287670 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:51.287938 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:51.287945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:51.288213 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:51.288336 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:51.288343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.289244 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.289249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:51.290182 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:51.290193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.290829 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.290835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:51.291026 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:51.291481 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:51.291496 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:51.293946 (+ 2450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 06:52:51.294330 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:51.287514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:43183 (local address 127.20.120.254:44571)
0621 06:52:51.287695 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:51.287703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:51.287720 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:52:51.287765 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:51.287775 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:51.287819 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:51.287911 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:51.288471 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.289121 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.290311 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.290687 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.290808 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:51.294110 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:43183: BlockingRecv error: recv got EOF from 127.20.120.195:43183 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260621 06:52:51.294517 30175 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 21 06:52:51 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:51 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:51.398926 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:51.390904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:60195)
0621 06:52:51.391408 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:51.391424 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:51.391531 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:51.391783 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:51.391786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:51.392330 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:51.392571 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:51.392581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.393620 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.393624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:51.394624 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:51.394636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.395275 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.395279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:51.395404 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:51.395949 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:51.395965 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:51.398710 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":275,"threads_started":1}
W20260621 06:52:51.399173 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:51.390991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:60195 (local address 127.20.120.254:44571)
0621 06:52:51.391162 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:51.391167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:51.391494 (+ 327us) server_negotiation.cc:407] Connection header received
0621 06:52:51.391611 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:51.391614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:51.391668 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:51.391741 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:51.392692 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.393501 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.394746 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.395144 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.395225 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:51.398922 (+ 3697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:60195: BlockingRecv error: recv got EOF from 127.20.120.193:60195 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 21 06:52:51 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:51 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:51.868666 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:51.859806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:41393)
0621 06:52:51.860345 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:51.860362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:51.860472 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:51.860774 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:51.860778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:51.861195 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:51.861444 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:51.861454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.862519 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.862524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:51.863639 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:51.863651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.864320 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.864325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:51.864441 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:51.865137 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:51.865154 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:51.868452 (+ 3298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":293,"threads_started":1}
W20260621 06:52:51.868928 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:51.859857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:41393 (local address 127.20.120.254:44571)
0621 06:52:51.860044 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:51.860048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:51.860433 (+ 385us) server_negotiation.cc:407] Connection header received
0621 06:52:51.860588 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:51.860591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:51.860649 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:51.860729 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:51.861581 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.862359 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.863773 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:51.864144 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:51.864231 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:51.868685 (+ 4454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:41393: BlockingRecv error: recv got EOF from 127.20.120.194:41393 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 21 06:52:52 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:52 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:52.302568 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:52.295026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:45061)
0621 06:52:52.295525 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:52.295547 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:52.295652 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:52.295911 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:52.295913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:52.296330 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:52.296506 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:52.296514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:52.297463 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:52.297466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:52.298403 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:52.298414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:52.299020 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:52.299024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:52.299138 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:52.299723 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:52.299739 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:52.302349 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":271,"threads_started":1}
W20260621 06:52:52.302747 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:52.295096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:45061 (local address 127.20.120.254:44571)
0621 06:52:52.295257 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:52.295261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:52.295617 (+ 356us) server_negotiation.cc:407] Connection header received
0621 06:52:52.295729 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:52.295732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:52.295788 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:52.295873 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:52.296631 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:52.297342 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:52.298526 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:52.298881 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:52.298969 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:52.302549 (+ 3580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:45061: BlockingRecv error: recv got EOF from 127.20.120.195:45061 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 21 06:52:52 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:52 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:52.407594 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:52.399884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:39527)
0621 06:52:52.400371 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:52.400387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:52.400498 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:52.400793 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:52.400796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:52.401159 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:52.401316 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:52.401324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:52.402356 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:52.402359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:52.403247 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:52.403257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:52.403903 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:52.403908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:52.404099 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:52.404688 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:52.404703 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:52.407359 (+ 2656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":291,"threads_started":1}
W20260621 06:52:52.407763 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:52.399919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:39527 (local address 127.20.120.254:44571)
0621 06:52:52.400141 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:52.400150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:52.400467 (+ 317us) server_negotiation.cc:407] Connection header received
0621 06:52:52.400593 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:52.400597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:52.400665 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:52.400741 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:52.401430 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:52.402236 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:52.403357 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:52.403766 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:52.403873 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:52.407524 (+ 3651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:39527: BlockingRecv error: recv got EOF from 127.20.120.193:39527 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 21 06:52:52 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:52 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:52.882319 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:52.869861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:50023)
0621 06:52:52.870512 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:52.870528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:52.870650 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:52.870941 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:52.870944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:52.871317 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:52.871479 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:52.871486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:52.872441 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:52.872444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:52.873314 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:52.873323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:52.874012 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:52.874016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:52.874119 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:52.879106 (+ 4987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:52.879124 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:52.881951 (+ 2827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":385,"threads_started":1}
W20260621 06:52:52.882393 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:52.869952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:50023 (local address 127.20.120.254:44571)
0621 06:52:52.870267 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:52.870272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:52.870619 (+ 347us) server_negotiation.cc:407] Connection header received
0621 06:52:52.870737 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:52.870742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:52.870811 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:52.870905 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:52.871608 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:52.872331 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:52.873426 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:52.873846 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:52.873990 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:52.882165 (+ 8175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:50023: BlockingRecv error: recv got EOF from 127.20.120.194:50023 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 21 06:52:53 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:53 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:53.312613 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:53.303567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:46673)
0621 06:52:53.304176 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:53.304195 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:53.304393 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:53.304776 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:53.304780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:53.305210 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:53.305395 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:53.305405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:53.306671 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:53.306679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:53.307638 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:53.307650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:53.308442 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:53.308449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:53.308635 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:53.309246 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:53.309265 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:53.312223 (+ 2958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":351,"threads_started":1}
W20260621 06:52:53.312747 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:53.303794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:46673 (local address 127.20.120.254:44571)
0621 06:52:53.304033 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:53.304042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:53.304307 (+ 265us) server_negotiation.cc:407] Connection header received
0621 06:52:53.304549 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:53.304555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:53.304619 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:53.304735 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:53.305572 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:53.306531 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:53.307793 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:53.308243 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:53.308414 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:53.312459 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:46673: BlockingRecv error: recv got EOF from 127.20.120.195:46673 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 21 06:52:53 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:53 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:53.416548 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:53.408465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:40479)
0621 06:52:53.408916 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:53.408933 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:53.409059 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:53.409321 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:53.409324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:53.409641 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:53.409797 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:53.409804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:53.410829 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:53.410833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:53.411848 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:53.411859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:53.412455 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:53.412459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:53.412582 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:53.413244 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:53.413262 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:53.416189 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
W20260621 06:52:53.416630 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:53.408531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:40479 (local address 127.20.120.254:44571)
0621 06:52:53.408766 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:53.408773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:53.409016 (+ 243us) server_negotiation.cc:407] Connection header received
0621 06:52:53.409143 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:53.409147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:53.409205 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:53.409282 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:53.409933 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:53.410708 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:53.411961 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:53.412337 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:53.412428 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:53.416374 (+ 3946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:40479: BlockingRecv error: recv got EOF from 127.20.120.193:40479 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 21 06:52:53 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:53 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:53.892222 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:53.883439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:42493)
0621 06:52:53.883970 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:53.883985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:53.884180 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:53.884517 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:53.884522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:53.884917 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:53.885092 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:53.885100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:53.886276 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:53.886281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:53.887621 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:53.887638 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:53.888372 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:53.888377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:53.888498 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:53.889135 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:53.889152 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:53.891924 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":299,"threads_started":1}
W20260621 06:52:53.892318 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:53.883519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:42493 (local address 127.20.120.254:44571)
0621 06:52:53.883746 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:53.883754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:53.884111 (+ 357us) server_negotiation.cc:407] Connection header received
0621 06:52:53.884309 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:53.884314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:53.884382 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:53.884488 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:53.885238 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:53.886117 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:53.887788 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:53.888244 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:53.888346 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:53.892135 (+ 3789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:42493: BlockingRecv error: recv got EOF from 127.20.120.194:42493 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 21 06:52:54 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:54 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:54.321643 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:54.313619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:51075)
0621 06:52:54.314282 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:54.314326 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:54.314462 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:54.314854 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:54.314860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:54.315343 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:54.315501 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:54.315517 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:54.316456 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:54.316459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:54.317293 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:54.317302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:54.317984 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:54.317990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:54.318128 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:54.318670 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:54.318686 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:54.321397 (+ 2711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":301,"threads_started":1}
W20260621 06:52:54.321808 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:54.313707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:51075 (local address 127.20.120.254:44571)
0621 06:52:54.313955 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:54.313962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:54.314410 (+ 448us) server_negotiation.cc:407] Connection header received
0621 06:52:54.314609 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:54.314614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:54.314697 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:54.314788 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:54.315630 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:54.316356 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:54.317401 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:54.317815 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:54.317920 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:54.321607 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:51075: BlockingRecv error: recv got EOF from 127.20.120.195:51075 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 21 06:52:54 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:54 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:54.424856 30189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:54.417388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:60535)
0621 06:52:54.417842 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:54.417857 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:54.417989 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:54.418246 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:54.418249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:54.418601 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:54.418730 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:54.418739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:54.419778 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:54.419784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:54.420838 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:54.420850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:54.421523 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:54.421529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:54.421701 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:54.422237 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:54.422252 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:54.424657 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":256,"threads_started":1}
W20260621 06:52:54.425040 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:54.417591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:60535 (local address 127.20.120.254:44571)
0621 06:52:54.417756 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:54.417760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:54.417942 (+ 182us) server_negotiation.cc:407] Connection header received
0621 06:52:54.418078 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:54.418082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:54.418134 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:54.418208 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:54.418893 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:54.419657 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:54.420970 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:54.421375 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:54.421502 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:54.424832 (+ 3330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:60535: BlockingRecv error: recv got EOF from 127.20.120.193:60535 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 06:52:54 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:54 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:54.901592 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:54.893179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:50955)
0621 06:52:54.893677 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:54.893692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:54.893844 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:54.894258 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:54.894263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:54.894641 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:54.894779 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:54.894786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:54.896039 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:54.896045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:54.897096 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:54.897108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:54.897823 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:54.897828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:54.898036 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:54.898600 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:54.898617 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:54.901239 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":301,"threads_started":1}
W20260621 06:52:54.901660 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:54.893222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:50955 (local address 127.20.120.254:44571)
0621 06:52:54.893436 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:54.893444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:54.893778 (+ 334us) server_negotiation.cc:407] Connection header received
0621 06:52:54.894014 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:54.894019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:54.894107 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:54.894213 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:54.894916 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:54.895937 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:54.897230 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:54.897674 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:54.897803 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:54.901435 (+ 3632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:50955: BlockingRecv error: recv got EOF from 127.20.120.194:50955 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 21 06:52:55 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:55 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:55.330585 30191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:55.322693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:58769)
0621 06:52:55.323183 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:55.323200 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:55.323308 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:55.323603 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:55.323606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:55.324009 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:55.324167 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:55.324176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:55.325190 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:55.325193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:55.326161 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:55.326171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:55.326798 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:55.326802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:55.326926 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:55.327577 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:55.327595 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:55.330353 (+ 2758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":276,"threads_started":1}
W20260621 06:52:55.330942 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:55.322713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:58769 (local address 127.20.120.254:44571)
0621 06:52:55.322877 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:55.322883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:55.323272 (+ 389us) server_negotiation.cc:407] Connection header received
0621 06:52:55.323413 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:55.323417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:55.323476 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:55.323562 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:55.324302 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:55.325079 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:55.326277 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:55.326681 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:55.326771 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:55.330646 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:58769: BlockingRecv error: recv got EOF from 127.20.120.195:58769 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 21 06:52:55 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:55 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:55.434468 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:55.425829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:47697)
0621 06:52:55.426399 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:55.426416 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:55.426584 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:55.426918 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:55.426922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:55.427365 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:55.427538 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:55.427548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:55.428665 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:55.428670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:55.429635 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:55.429646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:55.430510 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:55.430516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:55.430693 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:55.431227 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:55.431243 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:55.434117 (+ 2874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":310,"threads_started":1}
W20260621 06:52:55.434528 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:55.425883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:47697 (local address 127.20.120.254:44571)
0621 06:52:55.426159 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:55.426167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:55.426505 (+ 338us) server_negotiation.cc:407] Connection header received
0621 06:52:55.426723 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:55.426727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:55.426788 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:55.426890 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:55.427703 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:55.428527 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:55.429773 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:55.430363 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:55.430487 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:55.434306 (+ 3819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:47697: BlockingRecv error: recv got EOF from 127.20.120.193:47697 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 21 06:52:55 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:55 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:55.911077 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:55.902798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:37203)
0621 06:52:55.903278 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:55.903293 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:55.903397 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:55.903646 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:55.903650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:55.904055 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:55.904201 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:55.904208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:55.905268 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:55.905273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:55.906442 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:55.906455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:55.907234 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:55.907240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:55.907406 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:55.907946 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:55.907961 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:55.910792 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":204,"threads_started":1}
W20260621 06:52:55.911152 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:55.902995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:37203 (local address 127.20.120.254:44571)
0621 06:52:55.903231 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:55.903236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:55.903345 (+ 109us) server_negotiation.cc:407] Connection header received
0621 06:52:55.903473 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:55.903476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:55.903529 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:55.903604 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:55.904316 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:55.905117 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:55.906578 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:55.907092 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:55.907222 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:55.910955 (+ 3733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:37203: BlockingRecv error: recv got EOF from 127.20.120.194:37203 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 21 06:52:56 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:56 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:56.339480 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:56.331398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:41603)
0621 06:52:56.331921 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:56.331938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:56.332044 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:56.332335 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:56.332338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:56.332734 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:56.332885 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:56.332893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:56.333806 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:56.333809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:56.334703 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:56.334713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:56.335349 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:56.335352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:56.335470 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:56.336114 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:56.336152 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:56.339253 (+ 3101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":282,"threads_started":1}
W20260621 06:52:56.339608 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:56.331499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:41603 (local address 127.20.120.254:44571)
0621 06:52:56.331659 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:56.331663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:56.332009 (+ 346us) server_negotiation.cc:407] Connection header received
0621 06:52:56.332153 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:56.332157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:56.332215 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:56.332293 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:56.333004 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:56.333685 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:56.334822 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:56.335234 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:56.335323 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:56.339416 (+ 4093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:41603: BlockingRecv error: recv got EOF from 127.20.120.195:41603 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 21 06:52:56 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:56 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:56.443435 30195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:56.435465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:40775)
0621 06:52:56.435925 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:56.435942 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:56.436053 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:56.436371 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:56.436374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:56.436723 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:56.436862 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:56.436870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:56.437816 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:56.437819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:56.438728 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:56.438738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:56.439381 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:56.439385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:56.439503 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:56.440137 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:56.440155 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:56.443090 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":264,"threads_started":1}
W20260621 06:52:56.443600 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:56.435544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:40775 (local address 127.20.120.254:44571)
0621 06:52:56.435821 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:56.435828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:56.436025 (+ 197us) server_negotiation.cc:407] Connection header received
0621 06:52:56.436165 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:56.436168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:56.436253 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:56.436335 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:56.436979 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:56.437707 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:56.438845 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:56.439254 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:56.439345 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:56.443315 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:40775: BlockingRecv error: recv got EOF from 127.20.120.193:40775 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Jun 21 06:52:56 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:56 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:56.920054 30196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:56.912049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:60413)
0621 06:52:56.912636 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:56.912652 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:56.912764 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:56.913095 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:56.913100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:56.913463 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:56.913636 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:56.913644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:56.914653 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:56.914657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:56.915521 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:56.915531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:56.916168 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:56.916171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:56.916286 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:56.916933 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:56.916950 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:56.919710 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":299,"threads_started":1}
W20260621 06:52:56.920202 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:56.912126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:60413 (local address 127.20.120.254:44571)
0621 06:52:56.912330 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:56.912336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:56.912716 (+ 380us) server_negotiation.cc:407] Connection header received
0621 06:52:56.912854 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:56.912859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:56.912932 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:56.913033 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:56.913782 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:56.914531 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:56.915646 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:56.916046 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:56.916149 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:56.919948 (+ 3799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:60413: BlockingRecv error: recv got EOF from 127.20.120.194:60413 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 21 06:52:57 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:57 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:57.348649 30197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:57.340414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:48155)
0621 06:52:57.340957 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:57.340973 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:57.341138 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:57.341464 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:57.341468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:57.341845 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:57.342025 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:57.342043 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:57.343110 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:57.343114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:57.344095 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:57.344106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:57.344817 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:57.344822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:57.344994 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:57.345691 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:57.345708 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:57.348397 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":326,"threads_started":1}
W20260621 06:52:57.348877 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:57.340502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:48155 (local address 127.20.120.254:44571)
0621 06:52:57.340760 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:57.340768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:57.341058 (+ 290us) server_negotiation.cc:407] Connection header received
0621 06:52:57.341259 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:57.341265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:57.341333 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:57.341437 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:57.342180 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:57.342960 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:57.344229 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:57.344649 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:57.344802 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:57.348632 (+ 3830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:48155: BlockingRecv error: recv got EOF from 127.20.120.195:48155 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 21 06:52:57 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:57 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:57.452672 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:57.444570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:45651)
0621 06:52:57.445150 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:57.445172 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:57.445321 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:57.445936 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:57.445943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:57.446361 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:57.446504 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:57.446512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:57.447415 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:57.447418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:57.448295 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:57.448305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:57.448927 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:57.448931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:57.449040 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:57.449659 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:57.449676 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:57.452419 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":267,"threads_started":1}
W20260621 06:52:57.452826 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:57.444603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:45651 (local address 127.20.120.254:44571)
0621 06:52:57.444792 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:57.444798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:57.445239 (+ 441us) server_negotiation.cc:407] Connection header received
0621 06:52:57.445501 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:57.445507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:57.445584 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:57.445700 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:57.446627 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:57.447299 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:57.448410 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:57.448808 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:57.448895 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:57.452607 (+ 3712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:45651: BlockingRecv error: recv got EOF from 127.20.120.193:45651 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 21 06:52:57 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:57 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:57.932199 30199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:57.921046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:50623)
0621 06:52:57.921666 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:57.921683 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:57.921792 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:57.922182 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:57.922186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:57.922570 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:57.922723 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:57.922730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:57.923708 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:57.923712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:57.924744 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:57.924758 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:57.925672 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:57.925680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:57.925824 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:57.929155 (+ 3331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:57.929171 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:57.931936 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":285,"threads_started":1}
W20260621 06:52:57.932370 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:57.921142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:50623 (local address 127.20.120.254:44571)
0621 06:52:57.921358 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:57.921363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:57.921754 (+ 391us) server_negotiation.cc:407] Connection header received
0621 06:52:57.921884 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:57.921889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:57.922027 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:57.922111 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:57.922848 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:57.923586 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:57.924911 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:57.925526 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:57.925620 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:57.932121 (+ 6501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:50623: BlockingRecv error: recv got EOF from 127.20.120.194:50623 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 21 06:52:58 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:58 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:58.358872 30200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:58.349498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:42585)
0621 06:52:58.350383 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:58.350403 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:58.350613 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:58.350960 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:58.350963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:58.351334 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:58.351497 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:58.351506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:58.352890 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:58.352895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:58.353887 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:58.353900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:58.354764 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:58.354769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:58.354888 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:58.355478 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:58.355533 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:58.358636 (+ 3103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":520,"threads_started":1}
W20260621 06:52:58.359020 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:58.349527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:42585 (local address 127.20.120.254:44571)
0621 06:52:58.349723 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:58.349731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:58.350486 (+ 755us) server_negotiation.cc:407] Connection header received
0621 06:52:58.350750 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:58.350755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:58.350830 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:58.350907 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:58.351647 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:58.352733 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:58.354030 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:58.354625 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:58.354719 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:58.358802 (+ 4083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:42585: BlockingRecv error: recv got EOF from 127.20.120.195:42585 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 21 06:52:58 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:58 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:58.461484 30201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:58.453774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:39567)
0621 06:52:58.454353 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:58.454368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:58.454473 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:58.454923 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:58.454926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:58.455280 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:58.455418 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:58.455426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:58.456467 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:58.456471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:58.457428 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:58.457439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:58.458185 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:58.458190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:58.458295 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:58.458804 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:58.458820 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:58.461261 (+ 2441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":342,"threads_started":1}
W20260621 06:52:58.461642 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:58.453889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:39567 (local address 127.20.120.254:44571)
0621 06:52:58.454099 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:58.454107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:58.454428 (+ 321us) server_negotiation.cc:407] Connection header received
0621 06:52:58.454579 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:58.454584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:58.454721 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:58.454812 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:58.455529 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:58.456341 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:58.457537 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:58.458040 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:58.458136 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:58.461427 (+ 3291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:39567: BlockingRecv error: recv got EOF from 127.20.120.193:39567 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 21 06:52:58 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:58 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:58.941655 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:58.933101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:34833)
0621 06:52:58.933642 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:58.933663 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:58.933843 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:58.934217 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:58.934221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:58.934612 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:58.934754 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:58.934762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:58.935928 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:58.935934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:58.937012 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:58.937028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:58.937978 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:58.937986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:58.938157 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:58.938793 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:58.938810 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:58.941421 (+ 2611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":425,"threads_started":1}
W20260621 06:52:58.941821 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:58.933350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:34833 (local address 127.20.120.254:44571)
0621 06:52:58.933521 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:58.933527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:58.933768 (+ 241us) server_negotiation.cc:407] Connection header received
0621 06:52:58.934002 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:58.934009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:58.934075 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:58.934187 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:58.934897 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:58.935781 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:58.937152 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:58.937758 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:58.937886 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:58.941601 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:34833: BlockingRecv error: recv got EOF from 127.20.120.194:34833 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 21 06:52:59 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:59 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:59.367887 30203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:59.359801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:43483)
0621 06:52:59.360447 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:59.360462 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:59.360575 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:59.360852 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:59.360855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:59.361224 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:59.361372 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:59.361381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:59.362454 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:59.362457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:59.363400 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:59.363411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:59.364177 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:59.364180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:59.364280 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:59.364893 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:59.364909 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:59.367661 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":287,"threads_started":1}
W20260621 06:52:59.368046 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:59.359838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:43483 (local address 127.20.120.254:44571)
0621 06:52:59.360016 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:59.360022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:59.360513 (+ 491us) server_negotiation.cc:407] Connection header received
0621 06:52:59.360678 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:59.360681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:59.360735 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:59.360808 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:59.361490 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:59.362339 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:59.363503 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:59.364035 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:59.364129 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:59.367833 (+ 3704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:43483: BlockingRecv error: recv got EOF from 127.20.120.195:43483 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 21 06:52:59 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:59 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:59.470449 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:59.462555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:33889)
0621 06:52:59.463089 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:59.463114 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:59.463221 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:59.463551 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:59.463554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:59.463890 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:59.464042 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:59.464050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:59.464940 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:59.464944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:59.466054 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:59.466068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:59.466856 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:59.466862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:59.467017 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:59.467579 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:59.467595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:59.470213 (+ 2618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":294,"threads_started":1}
W20260621 06:52:59.470585 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:59.462590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:33889 (local address 127.20.120.254:44571)
0621 06:52:59.462746 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:59.462750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:59.463185 (+ 435us) server_negotiation.cc:407] Connection header received
0621 06:52:59.463311 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:59.463314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:59.463422 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:59.463502 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:59.464155 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:59.464834 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:59.466231 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:59.466700 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:59.466845 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:59.470385 (+ 3540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:33889: BlockingRecv error: recv got EOF from 127.20.120.193:33889 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 21 06:52:59 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:52:59 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:52:59.951220 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:52:59.942605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:37409)
0621 06:52:59.943126 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0621 06:52:59.943140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 06:52:59.943294 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:52:59.943656 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:52:59.943661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:52:59.944093 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:52:59.944248 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:59.944255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:59.945219 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:59.945224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:59.946236 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:52:59.946247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:52:59.946939 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:52:59.946944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:52:59.947109 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:59.947755 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:52:59.947772 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:52:59.950871 (+ 3099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":280,"threads_started":1}
W20260621 06:52:59.951315 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:52:59.942694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:37409 (local address 127.20.120.254:44571)
0621 06:52:59.942898 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 06:52:59.942907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:52:59.943222 (+ 315us) server_negotiation.cc:407] Connection header received
0621 06:52:59.943411 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:52:59.943416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:52:59.943483 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:52:59.943599 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:52:59.944381 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:59.945085 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:59.946377 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:52:59.946784 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:52:59.946909 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:52:59.951083 (+ 4174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:37409: BlockingRecv error: recv got EOF from 127.20.120.194:37409 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 21 06:53:00 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:00 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:00.376228 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:00.368819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:42381)
0621 06:53:00.369333 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:00.369349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:00.369451 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:00.369746 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:00.369749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:00.370122 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:00.370279 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:00.370287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:00.371225 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:00.371229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:00.372143 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:00.372153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:00.372763 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:00.372767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:00.372884 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:00.373440 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:00.373456 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:00.375941 (+ 2485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":302,"threads_started":1}
W20260621 06:53:00.376313 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:00.368908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:42381 (local address 127.20.120.254:44571)
0621 06:53:00.369075 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:00.369080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:00.369417 (+ 337us) server_negotiation.cc:407] Connection header received
0621 06:53:00.369541 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:00.369545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:00.369616 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:00.369699 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:00.370418 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:00.371120 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:00.372256 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:00.372646 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:00.372731 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:00.376121 (+ 3390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:42381: BlockingRecv error: recv got EOF from 127.20.120.195:42381 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 21 06:53:00 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:00 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:00.479417 30207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:00.471349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:40787)
0621 06:53:00.471899 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:00.471916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:00.472024 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:00.472325 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:00.472328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:00.472662 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:00.472797 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:00.472805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:00.474318 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:00.474326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:00.475210 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:00.475222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:00.475905 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:00.475909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:00.476019 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:00.476654 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:00.476670 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:00.479212 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":326,"threads_started":1}
W20260621 06:53:00.479597 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:00.471441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:40787 (local address 127.20.120.254:44571)
0621 06:53:00.471604 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:00.471608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:00.471984 (+ 376us) server_negotiation.cc:407] Connection header received
0621 06:53:00.472114 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:00.472117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:00.472197 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:00.472278 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:00.472963 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:00.474182 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:00.475371 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:00.475781 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:00.475874 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:00.479372 (+ 3498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:40787: BlockingRecv error: recv got EOF from 127.20.120.193:40787 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 21 06:53:00 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:00 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:00.960486 30208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:00.952275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:36817)
0621 06:53:00.952751 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:00.952768 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:00.952895 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:00.953178 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:00.953181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:00.953550 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:00.953697 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:00.953707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:00.954632 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:00.954635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:00.955675 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:00.955688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:00.956409 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:00.956414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:00.956531 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:00.957140 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:00.957158 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:00.960148 (+ 2990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":264,"threads_started":1}
W20260621 06:53:00.960565 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:00.952322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:36817 (local address 127.20.120.254:44571)
0621 06:53:00.952573 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:00.952579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:00.952852 (+ 273us) server_negotiation.cc:407] Connection header received
0621 06:53:00.952996 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:00.952999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:00.953057 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:00.953134 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:00.953818 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:00.954508 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:00.955817 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:00.956280 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:00.956366 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:00.960328 (+ 3962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:36817: BlockingRecv error: recv got EOF from 127.20.120.194:36817 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 21 06:53:01 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:01 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:01.385241 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:01.377204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:34329)
0621 06:53:01.377721 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:01.377739 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:01.377841 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:01.378180 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:01.378183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:01.378519 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:01.378664 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:01.378672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:01.379599 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:01.379602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:01.380534 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:01.380544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:01.381281 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:01.381286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:01.381425 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:01.382073 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:01.382092 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:01.384914 (+ 2822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":307,"threads_started":1}
W20260621 06:53:01.385326 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:01.377360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:34329 (local address 127.20.120.254:44571)
0621 06:53:01.377555 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:01.377560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:01.377808 (+ 248us) server_negotiation.cc:407] Connection header received
0621 06:53:01.377979 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:01.377983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:01.378040 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:01.378130 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:01.378794 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:01.379496 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:01.380654 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:01.381103 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:01.381197 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:01.385107 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:34329: BlockingRecv error: recv got EOF from 127.20.120.195:34329 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 06:53:01 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:01 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:01.488242 30210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:01.480262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:41083)
0621 06:53:01.480782 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:01.480799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:01.480956 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:01.481416 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:01.481422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:01.481898 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:01.482088 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:01.482097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:01.483165 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:01.483169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:01.484031 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:01.484041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:01.484685 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:01.484690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:01.484800 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:01.485323 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:01.485338 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:01.488006 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":292,"threads_started":1}
W20260621 06:53:01.488368 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:01.480365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:41083 (local address 127.20.120.254:44571)
0621 06:53:01.480547 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:01.480553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:01.480899 (+ 346us) server_negotiation.cc:407] Connection header received
0621 06:53:01.481136 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:01.481140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:01.481263 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:01.481361 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:01.482251 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:01.483017 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:01.484143 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:01.484562 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:01.484647 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:01.488178 (+ 3531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:41083: BlockingRecv error: recv got EOF from 127.20.120.193:41083 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 21 06:53:01 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:01 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:01.970077 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:01.961562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:50009)
0621 06:53:01.962111 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:01.962127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:01.962244 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:01.962553 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:01.962556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:01.962980 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:01.963172 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:01.963181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:01.964130 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:01.964134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:01.965031 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:01.965044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:01.965773 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:01.965780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:01.965970 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:01.966704 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:01.966730 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:01.969781 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":286,"threads_started":1}
W20260621 06:53:01.970242 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:01.961680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:50009 (local address 127.20.120.254:44571)
0621 06:53:01.961851 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:01.961855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:01.962199 (+ 344us) server_negotiation.cc:407] Connection header received
0621 06:53:01.962350 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:01.962354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:01.962412 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:01.962493 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:01.963303 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:01.964002 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:01.965160 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:01.965606 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:01.965693 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:01.970033 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:50009: BlockingRecv error: recv got EOF from 127.20.120.194:50009 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 21 06:53:02 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:02 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:02.394294 30212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:02.386232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:52613)
0621 06:53:02.386802 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:02.386819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:02.386926 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:02.387201 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:02.387204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:02.387567 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:02.387751 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:02.387759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:02.388698 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:02.388701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:02.389565 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:02.389574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:02.390317 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:02.390321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:02.390436 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:02.391000 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:02.391016 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:02.393944 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":319,"threads_started":1}
W20260621 06:53:02.394337 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:02.386477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:52613 (local address 127.20.120.254:44571)
0621 06:53:02.386689 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:02.386693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:02.386891 (+ 198us) server_negotiation.cc:407] Connection header received
0621 06:53:02.387018 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:02.387022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:02.387079 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:02.387158 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:02.387880 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:02.388592 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:02.389692 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:02.390175 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:02.390265 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:02.394112 (+ 3847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:52613: BlockingRecv error: recv got EOF from 127.20.120.195:52613 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 21 06:53:02 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:02 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:02.497817 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:02.489213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:55115)
0621 06:53:02.489723 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:02.489763 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:02.489968 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:02.490422 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:02.490426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:02.490948 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:02.491139 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:02.491148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:02.492286 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:02.492290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:02.493269 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:02.493285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:02.494192 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:02.494196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:02.494321 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:02.494889 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:02.494905 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:02.497577 (+ 2672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":278,"threads_started":1}
W20260621 06:53:02.498013 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:02.489308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:55115 (local address 127.20.120.254:44571)
0621 06:53:02.489507 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:02.489514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:02.489880 (+ 366us) server_negotiation.cc:407] Connection header received
0621 06:53:02.490109 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:02.490114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:02.490235 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:02.490346 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:02.491274 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:02.492138 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:02.493434 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:02.494074 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:02.494181 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:02.497776 (+ 3595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:55115: BlockingRecv error: recv got EOF from 127.20.120.193:55115 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 06:53:02 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:02 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:02.982262 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:02.971088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:35145)
0621 06:53:02.971586 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:02.971602 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:02.971756 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:02.972095 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:02.972099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:02.972516 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:02.972664 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:02.972672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:02.973628 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:02.973634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:02.974708 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:02.974720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:02.975422 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:02.975428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:02.975614 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:02.979367 (+ 3753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:02.979385 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:02.982026 (+ 2641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":295,"threads_started":1}
W20260621 06:53:02.982445 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:02.971160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:35145 (local address 127.20.120.254:44571)
0621 06:53:02.971371 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:02.971379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:02.971686 (+ 307us) server_negotiation.cc:407] Connection header received
0621 06:53:02.971889 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:02.971893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:02.971951 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:02.972052 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:02.972802 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:02.973488 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:02.974844 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:02.975282 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:02.975412 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:02.982216 (+ 6804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:35145: BlockingRecv error: recv got EOF from 127.20.120.194:35145 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 21 06:53:03 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:03 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:03.403640 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:03.395275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:58573)
0621 06:53:03.395721 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:03.395736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:03.395854 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:03.396145 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:03.396149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:03.396548 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:03.396695 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:03.396704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:03.397796 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:03.397806 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:03.398984 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:03.398997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:03.399687 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:03.399693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:03.399877 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:03.400525 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:03.400542 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:03.403289 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":260,"threads_started":1}
W20260621 06:53:03.403726 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:03.395309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:58573 (local address 127.20.120.254:44571)
0621 06:53:03.395526 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:03.395536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:03.395812 (+ 276us) server_negotiation.cc:407] Connection header received
0621 06:53:03.395952 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:03.395955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:03.396014 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:03.396102 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:03.396808 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:03.397639 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:03.399125 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:03.399544 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:03.399676 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:03.403484 (+ 3808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:58573: BlockingRecv error: recv got EOF from 127.20.120.195:58573 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 21 06:53:03 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:03 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:03.506938 30216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:03.498743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:59647)
0621 06:53:03.499273 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:03.499289 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:03.499417 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:03.499702 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:03.499705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:03.500075 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:03.500242 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:03.500250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:03.501183 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:03.501195 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:03.502198 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:03.502211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:03.502879 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:03.502883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:03.502994 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:03.503556 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:03.503572 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:03.506697 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":269,"threads_started":1}
W20260621 06:53:03.507120 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:03.498803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:59647 (local address 127.20.120.254:44571)
0621 06:53:03.498958 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:03.498962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:03.499359 (+ 397us) server_negotiation.cc:407] Connection header received
0621 06:53:03.499506 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:03.499509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:03.499578 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:03.499659 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:03.500363 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:03.501053 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:03.502314 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:03.502732 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:03.502823 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:03.506878 (+ 4055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:59647: BlockingRecv error: recv got EOF from 127.20.120.193:59647 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 21 06:53:03 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:03 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:03.991961 30217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:03.983257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:40947)
0621 06:53:03.983969 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:03.983984 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:03.984139 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:03.984523 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:03.984528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:03.985057 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:03.985268 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:03.985280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:03.986406 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:03.986412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:03.987329 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:03.987339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:03.988051 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:03.988057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:03.988219 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:03.988894 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:03.988910 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:03.991724 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":470,"threads_started":1}
W20260621 06:53:03.992383 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:03.983555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:40947 (local address 127.20.120.254:44571)
0621 06:53:03.983768 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:03.983776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:03.984068 (+ 292us) server_negotiation.cc:407] Connection header received
0621 06:53:03.984308 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:03.984311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:03.984385 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:03.984489 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:03.985390 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:03.986262 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:03.987467 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:03.987883 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:03.988017 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:03.991967 (+ 3950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:40947: BlockingRecv error: recv got EOF from 127.20.120.194:40947 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 21 06:53:04 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:04 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:04.412969 30218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:04.404652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:52617)
0621 06:53:04.405161 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:04.405178 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:04.405302 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:04.405586 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:04.405589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:04.405976 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:04.406140 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:04.406150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:04.407311 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:04.407314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:04.408251 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:04.408262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:04.408912 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:04.408916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:04.409027 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:04.409620 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:04.409647 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:04.412562 (+ 2915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":277,"threads_started":1}
W20260621 06:53:04.413005 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:04.404701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:52617 (local address 127.20.120.254:44571)
0621 06:53:04.404863 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:04.404868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:04.405251 (+ 383us) server_negotiation.cc:407] Connection header received
0621 06:53:04.405397 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:04.405400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:04.405455 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:04.405531 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:04.406270 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:04.407175 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:04.408382 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:04.408796 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:04.408889 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:04.412784 (+ 3895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:52617: BlockingRecv error: recv got EOF from 127.20.120.195:52617 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 21 06:53:04 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:04 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:04.516597 30219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:04.507979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:36233)
0621 06:53:04.508441 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:04.508466 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:04.508676 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:04.509029 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:04.509033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:04.509497 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:04.509729 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:04.509739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:04.510936 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:04.510944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:04.511908 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:04.511919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:04.512638 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:04.512644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:04.512815 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:04.513396 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:04.513413 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:04.516184 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":221,"threads_started":1}
W20260621 06:53:04.516660 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:04.508067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:36233 (local address 127.20.120.254:44571)
0621 06:53:04.508411 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:04.508417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:04.508576 (+ 159us) server_negotiation.cc:407] Connection header received
0621 06:53:04.508827 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:04.508832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:04.508902 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:04.509006 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:04.509864 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:04.510803 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:04.512048 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:04.512481 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:04.512611 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:04.516413 (+ 3802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:36233: BlockingRecv error: recv got EOF from 127.20.120.193:36233 (error 108)
Metrics: {"server-negotiator.queue_time_us":215}
Jun 21 06:53:04 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:04 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:05.000860 30220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:04.992864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:51691)
0621 06:53:04.993299 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:04.993316 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:04.993441 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:04.993776 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:04.993779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:04.994165 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:04.994335 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:04.994344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:04.995287 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:04.995290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:04.996298 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:04.996313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:04.997156 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:04.997162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:04.997340 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:04.997959 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:04.997976 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:05.000615 (+ 2639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":244,"threads_started":1}
W20260621 06:53:05.001068 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:04.992925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:51691 (local address 127.20.120.254:44571)
0621 06:53:04.993146 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:04.993153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:04.993410 (+ 257us) server_negotiation.cc:407] Connection header received
0621 06:53:04.993532 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:04.993536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:04.993606 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:04.993683 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:04.994452 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:04.995170 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:04.996449 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:04.997022 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:04.997150 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:05.000821 (+ 3671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:51691: BlockingRecv error: recv got EOF from 127.20.120.194:51691 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 21 06:53:05 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:05 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.195@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:05.421788 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:05.413880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.195:42401)
0621 06:53:05.414390 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:05.414406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:05.414518 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:05.414835 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:05.414839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:05.415276 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:05.415511 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:05.415525 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:05.416751 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:05.416756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:05.417578 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:05.417589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:05.418290 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:05.418294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:05.418409 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:05.418960 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:05.418976 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:05.421537 (+ 2561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":238,"threads_started":1}
W20260621 06:53:05.422030 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:05.414001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:42401 (local address 127.20.120.254:44571)
0621 06:53:05.414260 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:05.414268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:05.414489 (+ 221us) server_negotiation.cc:407] Connection header received
0621 06:53:05.414620 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:05.414624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:05.414685 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:05.414768 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:05.415663 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:05.416573 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:05.417687 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:05.418175 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:05.418266 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:05.421786 (+ 3520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.195:42401: BlockingRecv error: recv got EOF from 127.20.120.195:42401 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 06:53:05 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:05 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.193@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:05.526494 30222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:05.517892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.193:47551)
0621 06:53:05.518474 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:05.518490 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:05.518651 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:05.519034 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:05.519038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:05.519432 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:05.519591 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:05.519600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:05.520677 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:05.520684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:05.521598 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:05.521608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:05.522397 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:05.522402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:05.522581 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:05.523242 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:05.523258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:05.526205 (+ 2947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":285,"threads_started":1}
W20260621 06:53:05.526635 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:05.518062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:47551 (local address 127.20.120.254:44571)
0621 06:53:05.518299 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:05.518307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:05.518578 (+ 271us) server_negotiation.cc:407] Connection header received
0621 06:53:05.518776 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:05.518781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:05.518879 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:05.518987 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:05.519754 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:05.520524 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:05.521746 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:05.522243 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:05.522371 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:05.526423 (+ 4052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.193:47551: BlockingRecv error: recv got EOF from 127.20.120.193:47551 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 21 06:53:06 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 06:53:06 dist-test-slave-gmf2 krb5kdc[29178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.120.194@KRBTEST.COM for oryx/127.20.120.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 06:53:06.009984 30223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:06.002102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44571 (local address 127.20.120.194:45515)
0621 06:53:06.002622 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:06.002637 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:06.002752 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:06.003022 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:06.003025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:06.003395 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:06.003561 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:06.003569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:06.004427 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:06.004431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:06.005425 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:06.005446 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:06.006331 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:06.006336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:06.006476 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:06.007135 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:06.007162 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:06.009684 (+ 2522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.120.254:44571: Server oryx/127.20.120.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":282,"threads_started":1}
W20260621 06:53:06.010298 29914 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:06.002140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:45515 (local address 127.20.120.254:44571)
0621 06:53:06.002303 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:06.002307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:06.002706 (+ 399us) server_negotiation.cc:407] Connection header received
0621 06:53:06.002844 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:06.002848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:06.002902 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:06.002982 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:06.003668 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:06.004316 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:06.005574 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:06.006185 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:06.006277 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:06.009864 (+ 3587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.120.194:45515: BlockingRecv error: recv got EOF from 127.20.120.194:45515 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 06:53:06.277189 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 29765
I20260621 06:53:06.289086 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:06.278218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.20.120.193:40755)
0621 06:53:06.278738 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:06.278745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:06.278788 (+ 43us) server_negotiation.cc:407] Connection header received
0621 06:53:06.278844 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:06.278851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:06.279066 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:06.279236 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:06.280733 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:06.281687 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:06.283023 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:06.283554 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:06.283663 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:06.286140 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:06.286149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:06.286162 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:06.286195 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:06.288654 (+ 2459us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.120.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 06:53:06.288889 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45170: Request ticket server kudu/127.20.120.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":244,"threads_started":1}
W20260621 06:53:06.289201 30225 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45170: Request ticket server kudu/127.20.120.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 06:53:06.289242 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 29765: Not authorized: Client connection negotiation failed: client connection to 127.20.120.193:40755: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.120.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 06:53:06.289484 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 29765
I20260621 06:53:06.305799 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 29916
I20260621 06:53:06.317499 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:06.306943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.20.120.194:38167)
0621 06:53:06.307521 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:06.307528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:06.307557 (+ 29us) server_negotiation.cc:407] Connection header received
0621 06:53:06.307612 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:06.307618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:06.307832 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:06.307978 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:06.309112 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:06.310353 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:06.311580 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:06.312119 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:06.312241 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:06.314707 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:06.314715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:06.314729 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:06.314761 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:06.316971 (+ 2210us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.120.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 06:53:06.317212 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52942: Request ticket server kudu/127.20.120.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":298,"threads_started":1}
W20260621 06:53:06.317600 30226 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52942: Request ticket server kudu/127.20.120.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 06:53:06.317623 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 29916: Not authorized: Client connection negotiation failed: client connection to 127.20.120.194:38167: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.120.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 06:53:06.317811 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 29916
I20260621 06:53:06.333983 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 30047
I20260621 06:53:06.345592 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:06.334863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.20.120.195:33643)
0621 06:53:06.335415 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:06.335422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:06.335447 (+ 25us) server_negotiation.cc:407] Connection header received
0621 06:53:06.335504 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:06.335509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:06.335729 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:06.335877 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:06.337110 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:06.338257 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:06.339355 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:06.339847 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:06.339938 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:06.342408 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:06.342415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:06.342428 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:06.342459 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:06.344980 (+ 2521us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.120.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 06:53:06.345226 (+ 246us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45996: Request ticket server kudu/127.20.120.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":309,"threads_started":1}
W20260621 06:53:06.345606 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 30047: Not authorized: Client connection negotiation failed: client connection to 127.20.120.195:33643: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.120.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 06:53:06.345696 30227 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45996: Request ticket server kudu/127.20.120.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 06:53:06.345777 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 30047
I20260621 06:53:06.362790 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 29696
I20260621 06:53:06.364059 29758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:06.476511 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 29696
2026-06-21T06:53:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20962 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-21T06:53:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:53:06Z Disabled control of system clock
I20260621 06:53:06.549752 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:32953
--webserver_interface=127.20.120.254
--webserver_port=0
--builtin_ntp_servers=127.20.120.212:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:32953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 06:53:06.864248 30232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:06.864518 30232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:06.864593 30232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:06.871672 30232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:53:06.871790 30232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:06.871829 30232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:06.871896 30232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:53:06.871940 30232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:53:06.880712 30232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:32953
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:32953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:06.881953 30232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:06.883783 30232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:06.895480 30238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:06.895605 30240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:06.895741 30232 server_base.cc:1034] running on GCE node
W20260621 06:53:06.895641 30237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:06.896652 30232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:06.897898 30232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:06.899200 30232 hybrid_clock.cc:648] HybridClock initialized: now 1782024786899133 us; error 71 us; skew 500 ppm
I20260621 06:53:06.899552 30232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:53:06.902050 30232 webserver.cc:460] Webserver started at http://127.20.120.254:40527/ using document root <none> and password file <none>
I20260621 06:53:06.902751 30232 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:06.902884 30232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:06.903210 30232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:53:06.906081 30232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "8b7dd6b24e724198860fc255f0981fc3"
format_stamp: "Formatted at 2026-06-21 06:53:06 on dist-test-slave-gmf2"
server_key: "81a49d98d2a9c3b15f319f9db2a16d44"
server_key_iv: "48ebb755aa0fb4a954743f3bdfb72d5a"
server_key_version: "encryptionkey@0"
I20260621 06:53:06.906881 30232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "8b7dd6b24e724198860fc255f0981fc3"
format_stamp: "Formatted at 2026-06-21 06:53:06 on dist-test-slave-gmf2"
server_key: "81a49d98d2a9c3b15f319f9db2a16d44"
server_key_iv: "48ebb755aa0fb4a954743f3bdfb72d5a"
server_key_version: "encryptionkey@0"
I20260621 06:53:06.911830 30232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:53:06.915198 30246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:06.916500 30232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:53:06.916687 30232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "8b7dd6b24e724198860fc255f0981fc3"
format_stamp: "Formatted at 2026-06-21 06:53:06 on dist-test-slave-gmf2"
server_key: "81a49d98d2a9c3b15f319f9db2a16d44"
server_key_iv: "48ebb755aa0fb4a954743f3bdfb72d5a"
server_key_version: "encryptionkey@0"
I20260621 06:53:06.916903 30232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:06.940826 30232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:06.941679 30232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:06.942029 30232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:06.960005 30232 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:32953
I20260621 06:53:06.960027 30297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:32953 every 8 connection(s)
I20260621 06:53:06.961315 30232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:53:06.964831 30298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:53:06.968219 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 30232
I20260621 06:53:06.968425 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:53:06.969033 20963 external_mini_cluster.cc:1428] Setting key ab8eb7b2f883e99b751bb5b7988b476e
I20260621 06:53:06.975603 30298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3: Bootstrap starting.
I20260621 06:53:06.979120 30298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3: Neither blocks nor log segments found. Creating new log.
I20260621 06:53:06.980366 30298 log.cc:824] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3: Log is configured to *not* fsync() on all Append() calls
I20260621 06:53:06.983263 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:06.970990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.20.120.254:32953)
0621 06:53:06.971924 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:06.971934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:06.971975 (+ 41us) server_negotiation.cc:407] Connection header received
0621 06:53:06.974079 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:06.974102 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:06.974551 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:06.974889 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:06.975565 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:06.976884 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:06.978735 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:06.979455 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:06.979591 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:06.980768 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:06.980775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:06.980788 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:06.981566 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:06.981674 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:06.981708 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:06.981818 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:06.982074 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 06:53:06.982394 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":338,"threads_started":1}
I20260621 06:53:06.983542 30298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3: No bootstrap required, opened a new log
I20260621 06:53:06.991017 30298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7dd6b24e724198860fc255f0981fc3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32953 } }
I20260621 06:53:06.991436 30298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:53:06.991516 30298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7dd6b24e724198860fc255f0981fc3, State: Initialized, Role: FOLLOWER
I20260621 06:53:06.992054 30298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7dd6b24e724198860fc255f0981fc3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32953 } }
I20260621 06:53:06.992267 30298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:53:06.992359 30298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:53:06.992478 30298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:53:06.994599 30298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7dd6b24e724198860fc255f0981fc3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32953 } }
I20260621 06:53:06.995018 30298 leader_election.cc:302] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7dd6b24e724198860fc255f0981fc3; no voters:
I20260621 06:53:06.995517 30298 leader_election.cc:288] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:53:06.995662 30303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:53:06.996343 30303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7dd6b24e724198860fc255f0981fc3, State: Running, Role: LEADER
I20260621 06:53:06.996901 30303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7dd6b24e724198860fc255f0981fc3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32953 } }
I20260621 06:53:06.997484 30298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:53:06.999945 30305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7dd6b24e724198860fc255f0981fc3. Latest consensus state: current_term: 1 leader_uuid: "8b7dd6b24e724198860fc255f0981fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7dd6b24e724198860fc255f0981fc3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32953 } } }
I20260621 06:53:07.000375 30305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [sys.catalog]: This master's current role is: LEADER
I20260621 06:53:07.002041 30312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:53:07.002203 30304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b7dd6b24e724198860fc255f0981fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7dd6b24e724198860fc255f0981fc3" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32953 } } }
I20260621 06:53:07.003854 30304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3 [sys.catalog]: This master's current role is: LEADER
I20260621 06:53:07.007753 30312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:53:07.016963 30312 catalog_manager.cc:1347] Generated new cluster ID: 028653f261bb4f85919cf70b6b6fb287
I20260621 06:53:07.017086 30312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:53:07.037076 30312 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:53:07.038533 30312 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:53:07.064669 30312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b7dd6b24e724198860fc255f0981fc3: Generated new TSK 0
I20260621 06:53:07.065613 30312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:53:07.071350 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:32953
--builtin_ntp_servers=127.20.120.212:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260621 06:53:07.413584 30322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:07.413842 30322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:07.414013 30322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:07.421007 30322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:07.421190 30322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:07.421370 30322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:53:07.430289 30322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:32953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:07.431608 30322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:07.433319 30322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:07.449990 30328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:07.450037 30330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:07.450054 30327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:07.455307 30322 server_base.cc:1034] running on GCE node
I20260621 06:53:07.455921 30322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:07.456857 30322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:07.458132 30322 hybrid_clock.cc:648] HybridClock initialized: now 1782024787458093 us; error 30 us; skew 500 ppm
I20260621 06:53:07.458501 30322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:53:07.460794 30322 webserver.cc:460] Webserver started at http://127.20.120.193:32903/ using document root <none> and password file <none>
I20260621 06:53:07.461531 30322 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:07.461668 30322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:07.462076 30322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:53:07.464931 30322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "8fa9a9978b604e8b922d8e78bba84e3f"
format_stamp: "Formatted at 2026-06-21 06:53:07 on dist-test-slave-gmf2"
server_key: "a8dd244b9d76e08bf5ee65ccba7065aa"
server_key_iv: "8703709d21e36eb93145cf15cad77044"
server_key_version: "encryptionkey@0"
I20260621 06:53:07.465752 30322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "8fa9a9978b604e8b922d8e78bba84e3f"
format_stamp: "Formatted at 2026-06-21 06:53:07 on dist-test-slave-gmf2"
server_key: "a8dd244b9d76e08bf5ee65ccba7065aa"
server_key_iv: "8703709d21e36eb93145cf15cad77044"
server_key_version: "encryptionkey@0"
I20260621 06:53:07.470956 30322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260621 06:53:07.474318 30336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:07.475579 30322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 06:53:07.475757 30322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "8fa9a9978b604e8b922d8e78bba84e3f"
format_stamp: "Formatted at 2026-06-21 06:53:07 on dist-test-slave-gmf2"
server_key: "a8dd244b9d76e08bf5ee65ccba7065aa"
server_key_iv: "8703709d21e36eb93145cf15cad77044"
server_key_version: "encryptionkey@0"
I20260621 06:53:07.475980 30322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:07.505353 30322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:07.506263 30322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:07.506551 30322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:07.507447 30322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:53:07.508951 30322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:53:07.509074 30322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:07.509194 30322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:53:07.509277 30322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:07.542455 30322 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:39595
I20260621 06:53:07.542562 30448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:39595 every 8 connection(s)
I20260621 06:53:07.543771 30322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:53:07.546703 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 30322
I20260621 06:53:07.546900 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:53:07.547801 20963 external_mini_cluster.cc:1428] Setting key 82f70e61b75ccaa1dfc44fe6905a4f80
I20260621 06:53:07.551043 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:32953
--builtin_ntp_servers=127.20.120.212:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 06:53:07.558090 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:07.546352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:42311 (local address 127.20.120.254:32953)
0621 06:53:07.546821 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:07.546828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:07.548264 (+ 1436us) server_negotiation.cc:407] Connection header received
0621 06:53:07.550474 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:07.550478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:07.550572 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:07.550668 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:07.552109 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:07.552841 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:07.555283 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:07.555656 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:07.555762 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:07.557334 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:07.557339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:07.557342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:07.557358 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:07.557411 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:07.557422 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:07.557486 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:07.557734 (+ 248us) server_negotiation.cc:299] Negotiation successful
0621 06:53:07.557804 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":252,"threads_started":1}
I20260621 06:53:07.559173 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:07.546921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:32953 (local address 127.20.120.193:42311)
0621 06:53:07.548096 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:07.548143 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:07.550152 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:07.550949 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:07.550960 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:07.551088 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:07.551927 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:07.551956 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:07.552997 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:07.553004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:07.555146 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:07.555163 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:07.555896 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:07.555905 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:07.556228 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:07.556971 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:53:07.556977 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:07.557037 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:53:07.557135 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:53:07.557513 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:53:07.557525 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:53:07.557535 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 06:53:07.557723 (+ 188us) client_negotiation.cc:240] Negotiation successful
0621 06:53:07.558185 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":284,"threads_started":1}
I20260621 06:53:07.560593 30449 heartbeater.cc:347] Connected to a master server at 127.20.120.254:32953
I20260621 06:53:07.560896 30449 heartbeater.cc:464] Registering TS with master...
I20260621 06:53:07.561645 30449 heartbeater.cc:511] Master 127.20.120.254:32953 requested a full tablet report, sending...
I20260621 06:53:07.563803 30263 ts_manager.cc:194] Registered new tserver with Master: 8fa9a9978b604e8b922d8e78bba84e3f (127.20.120.193:39595)
I20260621 06:53:07.565429 30263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.193:42311
W20260621 06:53:07.867282 30453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:07.867553 30453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:07.867633 30453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:07.874823 30453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:07.874924 30453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:07.875105 30453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:53:07.884299 30453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:32953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:07.885623 30453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:07.887413 30453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:07.900686 30458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:07.900638 30461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:07.900655 30459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:07.902070 30453 server_base.cc:1034] running on GCE node
I20260621 06:53:07.902663 30453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:07.903738 30453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:07.904951 30453 hybrid_clock.cc:648] HybridClock initialized: now 1782024787904886 us; error 59 us; skew 500 ppm
I20260621 06:53:07.905254 30453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:53:07.907423 30453 webserver.cc:460] Webserver started at http://127.20.120.194:33097/ using document root <none> and password file <none>
I20260621 06:53:07.908141 30453 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:07.908267 30453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:07.908586 30453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:53:07.911396 30453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "c3787a6c4d8945e293854499b7dc8ed0"
format_stamp: "Formatted at 2026-06-21 06:53:07 on dist-test-slave-gmf2"
server_key: "c9c489a9c5fac6939e98918611fd998f"
server_key_iv: "cb3841a1ccdec51a5780a197f652c5bb"
server_key_version: "encryptionkey@0"
I20260621 06:53:07.912184 30453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "c3787a6c4d8945e293854499b7dc8ed0"
format_stamp: "Formatted at 2026-06-21 06:53:07 on dist-test-slave-gmf2"
server_key: "c9c489a9c5fac6939e98918611fd998f"
server_key_iv: "cb3841a1ccdec51a5780a197f652c5bb"
server_key_version: "encryptionkey@0"
I20260621 06:53:07.917341 30453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 06:53:07.920804 30467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:07.922084 30453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:53:07.922252 30453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "c3787a6c4d8945e293854499b7dc8ed0"
format_stamp: "Formatted at 2026-06-21 06:53:07 on dist-test-slave-gmf2"
server_key: "c9c489a9c5fac6939e98918611fd998f"
server_key_iv: "cb3841a1ccdec51a5780a197f652c5bb"
server_key_version: "encryptionkey@0"
I20260621 06:53:07.922456 30453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:07.943058 30453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:07.943812 30453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:07.944152 30453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:07.945050 30453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:53:07.946537 30453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:53:07.946655 30453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:07.946772 30453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:53:07.946868 30453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:07.979031 30453 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:42715
I20260621 06:53:07.979159 30579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:42715 every 8 connection(s)
I20260621 06:53:07.980611 30453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:53:07.985049 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 30453
I20260621 06:53:07.985219 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:53:07.985747 20963 external_mini_cluster.cc:1428] Setting key e3eea383efd0ecb9b4b2bbac3bd7b3a5
I20260621 06:53:07.988816 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:32953
--builtin_ntp_servers=127.20.120.212:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 06:53:07.998883 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:07.983193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:60099 (local address 127.20.120.254:32953)
0621 06:53:07.983351 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:07.983355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:07.984914 (+ 1559us) server_negotiation.cc:407] Connection header received
0621 06:53:07.987161 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:07.987165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:07.987224 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:07.987306 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:07.988612 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:07.989604 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:07.991910 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:07.992295 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:07.992388 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:07.997877 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:07.997886 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:07.997894 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:07.997939 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:07.997978 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:07.997996 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:07.998119 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:07.998382 (+ 263us) server_negotiation.cc:299] Negotiation successful
0621 06:53:07.998488 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 06:53:07.999812 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:07.983817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:32953 (local address 127.20.120.194:60099)
0621 06:53:07.984757 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:07.984802 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:07.986889 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:07.987551 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:07.987558 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:07.987655 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:07.988412 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:07.988438 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:07.989756 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:07.989763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:07.991757 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:07.991777 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:07.992508 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:07.992517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:07.992830 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:07.997543 (+ 4713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:53:07.997545 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:07.997621 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:53:07.997709 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:53:07.998117 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:53:07.998124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:53:07.998161 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 06:53:07.998314 (+ 153us) client_negotiation.cc:240] Negotiation successful
0621 06:53:07.998595 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":283,"threads_started":1}
I20260621 06:53:08.001133 30580 heartbeater.cc:347] Connected to a master server at 127.20.120.254:32953
I20260621 06:53:08.001447 30580 heartbeater.cc:464] Registering TS with master...
I20260621 06:53:08.002396 30580 heartbeater.cc:511] Master 127.20.120.254:32953 requested a full tablet report, sending...
I20260621 06:53:08.003858 30263 ts_manager.cc:194] Registered new tserver with Master: c3787a6c4d8945e293854499b7dc8ed0 (127.20.120.194:42715)
I20260621 06:53:08.004626 30263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.194:60099
W20260621 06:53:08.306579 30583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:08.306852 30583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:08.307005 30583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:08.314218 30583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:08.314314 30583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:08.314488 30583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:53:08.323789 30583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:32953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:08.325094 30583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:08.326985 30583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:08.340157 30589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:08.340196 30588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:08.340417 30591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:08.340991 30583 server_base.cc:1034] running on GCE node
I20260621 06:53:08.341516 30583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:08.342463 30583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:08.343659 30583 hybrid_clock.cc:648] HybridClock initialized: now 1782024788343595 us; error 58 us; skew 500 ppm
I20260621 06:53:08.343963 30583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:53:08.346283 30583 webserver.cc:460] Webserver started at http://127.20.120.195:37575/ using document root <none> and password file <none>
I20260621 06:53:08.346989 30583 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:08.347102 30583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:08.347430 30583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:53:08.350555 30583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "b05b2b0fd8c84d6a8d063fbc9c6d8f5c"
format_stamp: "Formatted at 2026-06-21 06:53:08 on dist-test-slave-gmf2"
server_key: "7b63f649125d0d628232ff363e07b087"
server_key_iv: "e2d920519c36ff33c0d9b7531c54fb4d"
server_key_version: "encryptionkey@0"
I20260621 06:53:08.351346 30583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "b05b2b0fd8c84d6a8d063fbc9c6d8f5c"
format_stamp: "Formatted at 2026-06-21 06:53:08 on dist-test-slave-gmf2"
server_key: "7b63f649125d0d628232ff363e07b087"
server_key_iv: "e2d920519c36ff33c0d9b7531c54fb4d"
server_key_version: "encryptionkey@0"
I20260621 06:53:08.356345 30583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:53:08.359831 30597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:08.361223 30583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:53:08.361424 30583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "b05b2b0fd8c84d6a8d063fbc9c6d8f5c"
format_stamp: "Formatted at 2026-06-21 06:53:08 on dist-test-slave-gmf2"
server_key: "7b63f649125d0d628232ff363e07b087"
server_key_iv: "e2d920519c36ff33c0d9b7531c54fb4d"
server_key_version: "encryptionkey@0"
I20260621 06:53:08.361660 30583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:08.378396 30583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:08.379379 30583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:08.379710 30583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:08.380757 30583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:53:08.382360 30583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:53:08.382503 30583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:08.382639 30583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:53:08.382735 30583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:08.417605 30583 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:39037
I20260621 06:53:08.417642 30709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:39037 every 8 connection(s)
I20260621 06:53:08.419103 30583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:53:08.422595 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 30583
I20260621 06:53:08.422752 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:53:08.423381 20963 external_mini_cluster.cc:1428] Setting key 5149dc6338772748a818d51c142d9aad
I20260621 06:53:08.432571 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:08.421990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:44853 (local address 127.20.120.254:32953)
0621 06:53:08.422235 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:08.422242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:08.423977 (+ 1735us) server_negotiation.cc:407] Connection header received
0621 06:53:08.426436 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:08.426444 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:08.426539 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:08.426645 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:08.427766 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:08.428528 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:08.430059 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:08.430485 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:08.430591 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:08.431822 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:08.431828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:08.431831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:08.431849 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:08.431881 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:08.431894 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:08.431981 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:08.432229 (+ 248us) server_negotiation.cc:299] Negotiation successful
0621 06:53:08.432304 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:53:08.433480 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:08.422609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:32953 (local address 127.20.120.195:44853)
0621 06:53:08.423809 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:08.423859 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:08.426098 (+ 2239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:08.426933 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:08.426941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:08.427027 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:08.427608 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:08.427624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:08.428661 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:08.428664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:08.429871 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:08.429882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:08.430670 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:08.430676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:08.430878 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:08.431516 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:53:08.431518 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:08.431556 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:53:08.431630 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:53:08.431981 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:53:08.431988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:53:08.431993 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 06:53:08.432128 (+ 135us) client_negotiation.cc:240] Negotiation successful
0621 06:53:08.432432 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":350,"threads_started":1}
I20260621 06:53:08.434859 30710 heartbeater.cc:347] Connected to a master server at 127.20.120.254:32953
I20260621 06:53:08.435178 30710 heartbeater.cc:464] Registering TS with master...
I20260621 06:53:08.435902 30710 heartbeater.cc:511] Master 127.20.120.254:32953 requested a full tablet report, sending...
I20260621 06:53:08.437243 30263 ts_manager.cc:194] Registered new tserver with Master: b05b2b0fd8c84d6a8d063fbc9c6d8f5c (127.20.120.195:39037)
I20260621 06:53:08.438218 30263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.195:44853
I20260621 06:53:08.439302 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 06:53:08.445549 30451 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:08.443823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.20.120.254:32953)
0621 06:53:08.444039 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:08.444049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:08.444488 (+ 439us) server_negotiation.cc:407] Connection header received
0621 06:53:08.444720 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:08.444725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:08.444799 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:08.444911 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:08.445340 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44832: BlockingRecv error: recv got EOF from 127.0.0.1:44832 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 06:53:08.446271 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 30322
I20260621 06:53:08.454157 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:08.447237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.20.120.193:39595)
0621 06:53:08.447766 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:08.447776 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:08.447815 (+ 39us) server_negotiation.cc:407] Connection header received
0621 06:53:08.447882 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:08.447888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:08.448114 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:08.448248 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:08.448539 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:08.449623 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:08.451076 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:08.451647 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:08.451752 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:08.452682 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:08.452690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:08.452703 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:08.453308 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:08.453356 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:08.453387 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:08.453473 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:08.453567 (+ 94us) server_negotiation.cc:299] Negotiation successful
0621 06:53:08.453677 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":283,"threads_started":1}
I20260621 06:53:08.455382 30444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:08.536288 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 30322
I20260621 06:53:08.552914 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 30453
I20260621 06:53:08.560369 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:08.553725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.20.120.194:42715)
0621 06:53:08.554289 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:08.554297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:08.554325 (+ 28us) server_negotiation.cc:407] Connection header received
0621 06:53:08.554384 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:08.554391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:08.554630 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:08.554777 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:08.555190 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:08.556150 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:08.557365 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:08.557976 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:08.558077 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:08.559080 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:08.559087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:08.559101 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:08.559687 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:08.559734 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:08.559752 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:08.559838 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:08.559961 (+ 123us) server_negotiation.cc:299] Negotiation successful
0621 06:53:08.560078 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":316,"threads_started":1}
I20260621 06:53:08.561720 30575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:08.642282 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 30453
I20260621 06:53:08.658762 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 30583
I20260621 06:53:08.667157 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:08.659724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.20.120.195:39037)
0621 06:53:08.660400 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:08.660406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:08.660436 (+ 30us) server_negotiation.cc:407] Connection header received
0621 06:53:08.660498 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:08.660504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:08.660725 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:08.660909 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:08.661360 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:08.662443 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:08.663760 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:08.664335 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:08.664494 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:08.665615 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:08.665624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:08.665639 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:08.666292 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:08.666366 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:08.666385 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:08.666510 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:08.666665 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 06:53:08.666804 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":383,"threads_started":1}
I20260621 06:53:08.668395 30705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:08.752128 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 30583
I20260621 06:53:08.768352 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 30232
I20260621 06:53:08.769599 30293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:08.870988 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 30232
2026-06-21T06:53:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2390 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-21T06:53:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:53:08Z Disabled control of system clock
I20260621 06:53:08.943027 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:35639
--webserver_interface=127.20.120.254
--webserver_port=0
--builtin_ntp_servers=127.20.120.212:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:35639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 06:53:09.268251 30729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:09.268582 30729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:09.268668 30729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:09.276065 30729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:53:09.276197 30729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:09.276234 30729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:09.276291 30729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:53:09.276335 30729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:53:09.285676 30729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:35639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:35639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:09.286994 30729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:09.288738 30729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:09.300027 30734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:09.300138 30735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:09.300202 30737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:09.301198 30729 server_base.cc:1034] running on GCE node
I20260621 06:53:09.301899 30729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:09.303246 30729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:09.304469 30729 hybrid_clock.cc:648] HybridClock initialized: now 1782024789304430 us; error 51 us; skew 500 ppm
I20260621 06:53:09.304811 30729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:53:09.306973 30729 webserver.cc:460] Webserver started at http://127.20.120.254:39447/ using document root <none> and password file <none>
I20260621 06:53:09.307624 30729 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:09.307756 30729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:09.308071 30729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:53:09.310909 30729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "db8b67c50d07489b8ab9b4b7e89ff9c2"
format_stamp: "Formatted at 2026-06-21 06:53:09 on dist-test-slave-gmf2"
server_key: "eaa74515044c948ac557d4dfbcb61082"
server_key_iv: "89e900aaacf990b37300f5f345558b17"
server_key_version: "encryptionkey@0"
I20260621 06:53:09.311671 30729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "db8b67c50d07489b8ab9b4b7e89ff9c2"
format_stamp: "Formatted at 2026-06-21 06:53:09 on dist-test-slave-gmf2"
server_key: "eaa74515044c948ac557d4dfbcb61082"
server_key_iv: "89e900aaacf990b37300f5f345558b17"
server_key_version: "encryptionkey@0"
I20260621 06:53:09.316624 30729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260621 06:53:09.319895 30743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:09.321187 30729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:53:09.321355 30729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "db8b67c50d07489b8ab9b4b7e89ff9c2"
format_stamp: "Formatted at 2026-06-21 06:53:09 on dist-test-slave-gmf2"
server_key: "eaa74515044c948ac557d4dfbcb61082"
server_key_iv: "89e900aaacf990b37300f5f345558b17"
server_key_version: "encryptionkey@0"
I20260621 06:53:09.321596 30729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:09.327191 30729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:09.327941 30729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:09.328212 30729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:09.345526 30729 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:35639
I20260621 06:53:09.345603 30794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:35639 every 8 connection(s)
I20260621 06:53:09.346869 30729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:53:09.350716 30795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:53:09.355279 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 30729
I20260621 06:53:09.355504 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:53:09.356115 20963 external_mini_cluster.cc:1428] Setting key c08d6f3f2e66bea0ef7dfef5969c3aa8
I20260621 06:53:09.361754 30795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2: Bootstrap starting.
I20260621 06:53:09.364547 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:09.358177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.20.120.254:35639)
0621 06:53:09.359241 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:09.359252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:09.359295 (+ 43us) server_negotiation.cc:407] Connection header received
0621 06:53:09.360903 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:09.360924 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:09.361339 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:09.361736 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:09.362882 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:09.362888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:09.362900 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:09.362913 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:09.362973 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:09.362989 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:09.363242 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:09.363402 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 06:53:09.363637 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":355,"threads_started":1}
I20260621 06:53:09.365446 30795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2: Neither blocks nor log segments found. Creating new log.
I20260621 06:53:09.366645 30795 log.cc:824] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2: Log is configured to *not* fsync() on all Append() calls
I20260621 06:53:09.369782 30795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2: No bootstrap required, opened a new log
I20260621 06:53:09.376936 30795 raft_consensus.cc:348] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8b67c50d07489b8ab9b4b7e89ff9c2" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 35639 } }
I20260621 06:53:09.377315 30795 raft_consensus.cc:374] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:53:09.377410 30795 raft_consensus.cc:729] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db8b67c50d07489b8ab9b4b7e89ff9c2, State: Initialized, Role: FOLLOWER
I20260621 06:53:09.378050 30795 consensus_queue.cc:260] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8b67c50d07489b8ab9b4b7e89ff9c2" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 35639 } }
I20260621 06:53:09.378289 30795 raft_consensus.cc:388] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:53:09.378445 30795 raft_consensus.cc:482] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:53:09.378618 30795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:53:09.380684 30795 raft_consensus.cc:504] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8b67c50d07489b8ab9b4b7e89ff9c2" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 35639 } }
I20260621 06:53:09.381170 30795 leader_election.cc:302] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db8b67c50d07489b8ab9b4b7e89ff9c2; no voters:
I20260621 06:53:09.381772 30795 leader_election.cc:288] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:53:09.381906 30800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:53:09.382717 30800 raft_consensus.cc:686] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [term 1 LEADER]: Becoming Leader. State: Replica: db8b67c50d07489b8ab9b4b7e89ff9c2, State: Running, Role: LEADER
I20260621 06:53:09.383248 30800 consensus_queue.cc:237] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8b67c50d07489b8ab9b4b7e89ff9c2" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 35639 } }
I20260621 06:53:09.384341 30795 sys_catalog.cc:564] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:53:09.386560 30801 sys_catalog.cc:455] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db8b67c50d07489b8ab9b4b7e89ff9c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8b67c50d07489b8ab9b4b7e89ff9c2" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 35639 } } }
I20260621 06:53:09.387137 30801 sys_catalog.cc:458] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [sys.catalog]: This master's current role is: LEADER
I20260621 06:53:09.386653 30802 sys_catalog.cc:455] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db8b67c50d07489b8ab9b4b7e89ff9c2. Latest consensus state: current_term: 1 leader_uuid: "db8b67c50d07489b8ab9b4b7e89ff9c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8b67c50d07489b8ab9b4b7e89ff9c2" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 35639 } } }
I20260621 06:53:09.387693 30802 sys_catalog.cc:458] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2 [sys.catalog]: This master's current role is: LEADER
I20260621 06:53:09.390967 30809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:53:09.395623 30809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:53:09.403524 30809 catalog_manager.cc:1347] Generated new cluster ID: 8b8e33659374475abf92c9a913d60e0c
I20260621 06:53:09.403641 30809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:53:09.421350 30809 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:53:09.421731 30809 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:53:09.427246 30809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db8b67c50d07489b8ab9b4b7e89ff9c2: Generated new TSK 0
I20260621 06:53:09.427922 30809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:53:09.437970 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:35639
--builtin_ntp_servers=127.20.120.212:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 06:53:09.765391 30819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:09.765681 30819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:09.765847 30819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:09.773366 30819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:09.773485 30819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:09.773690 30819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:53:09.783218 30819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:35639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:09.784664 30819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:09.786722 30819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:09.799569 30824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:09.799579 30825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:09.799625 30827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:09.800609 30819 server_base.cc:1034] running on GCE node
I20260621 06:53:09.801144 30819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:09.802021 30819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:09.803215 30819 hybrid_clock.cc:648] HybridClock initialized: now 1782024789803184 us; error 34 us; skew 500 ppm
I20260621 06:53:09.803535 30819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:53:09.805621 30819 webserver.cc:460] Webserver started at http://127.20.120.193:41847/ using document root <none> and password file <none>
I20260621 06:53:09.806345 30819 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:09.806460 30819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:09.806795 30819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:53:09.809530 30819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "ad228e04bb254627831e69e3e0d41338"
format_stamp: "Formatted at 2026-06-21 06:53:09 on dist-test-slave-gmf2"
server_key: "b02d7bfec3fdf2b4d496253db839ce55"
server_key_iv: "71a6f0cf21e9b76965ca910b5adbaec4"
server_key_version: "encryptionkey@0"
I20260621 06:53:09.810343 30819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "ad228e04bb254627831e69e3e0d41338"
format_stamp: "Formatted at 2026-06-21 06:53:09 on dist-test-slave-gmf2"
server_key: "b02d7bfec3fdf2b4d496253db839ce55"
server_key_iv: "71a6f0cf21e9b76965ca910b5adbaec4"
server_key_version: "encryptionkey@0"
I20260621 06:53:09.815202 30819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260621 06:53:09.818501 30833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:09.819882 30819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:53:09.820048 30819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "ad228e04bb254627831e69e3e0d41338"
format_stamp: "Formatted at 2026-06-21 06:53:09 on dist-test-slave-gmf2"
server_key: "b02d7bfec3fdf2b4d496253db839ce55"
server_key_iv: "71a6f0cf21e9b76965ca910b5adbaec4"
server_key_version: "encryptionkey@0"
I20260621 06:53:09.820278 30819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:09.826243 30819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:09.827032 30819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:09.827309 30819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:09.828248 30819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:53:09.829694 30819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:53:09.829828 30819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:09.830022 30819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:53:09.830114 30819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:09.864194 30819 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:45727
I20260621 06:53:09.864241 30945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:45727 every 8 connection(s)
I20260621 06:53:09.865747 30819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:53:09.873572 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 30819
I20260621 06:53:09.873831 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:53:09.874264 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:09.868462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:47685 (local address 127.20.120.254:35639)
0621 06:53:09.868916 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:09.868923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:09.870323 (+ 1400us) server_negotiation.cc:407] Connection header received
0621 06:53:09.872039 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:09.872042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:09.872095 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:09.872180 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:09.873470 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:09.873476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:09.873481 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:09.873527 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:09.873566 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:09.873578 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:09.873720 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:09.873933 (+ 213us) server_negotiation.cc:299] Negotiation successful
0621 06:53:09.873998 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":205,"threads_started":1}
I20260621 06:53:09.874537 20963 external_mini_cluster.cc:1428] Setting key 9a0751d4e9d7d89efebc0f179213e47f
I20260621 06:53:09.875324 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:09.869143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:35639 (local address 127.20.120.193:47685)
0621 06:53:09.870164 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:09.870211 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:09.871764 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:09.872412 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:09.872420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:09.872515 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:09.873163 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:53:09.873166 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:09.873203 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:53:09.873287 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:53:09.873703 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:53:09.873713 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:53:09.873720 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 06:53:09.874020 (+ 300us) client_negotiation.cc:240] Negotiation successful
0621 06:53:09.874265 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":275,"threads_started":1}
I20260621 06:53:09.876904 30946 heartbeater.cc:347] Connected to a master server at 127.20.120.254:35639
I20260621 06:53:09.877189 30946 heartbeater.cc:464] Registering TS with master...
I20260621 06:53:09.877702 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:35639
--builtin_ntp_servers=127.20.120.212:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 06:53:09.877730 30946 heartbeater.cc:511] Master 127.20.120.254:35639 requested a full tablet report, sending...
I20260621 06:53:09.879676 30760 ts_manager.cc:194] Registered new tserver with Master: ad228e04bb254627831e69e3e0d41338 (127.20.120.193:45727)
W20260621 06:53:10.201838 30950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:10.202178 30950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:10.202354 30950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:10.209386 30950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:10.209507 30950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:10.209673 30950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:53:10.219254 30950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:35639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:10.220546 30950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:10.222353 30950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:10.235481 30956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:10.235417 30955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:10.235666 30958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:10.236091 30950 server_base.cc:1034] running on GCE node
I20260621 06:53:10.236727 30950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:10.237660 30950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:10.238870 30950 hybrid_clock.cc:648] HybridClock initialized: now 1782024790238805 us; error 65 us; skew 500 ppm
I20260621 06:53:10.239189 30950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:53:10.241384 30950 webserver.cc:460] Webserver started at http://127.20.120.194:45793/ using document root <none> and password file <none>
I20260621 06:53:10.242146 30950 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:10.242267 30950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:10.242590 30950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:53:10.245352 30950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "c3b953cbf08a4fb8b113bd8ccff060a4"
format_stamp: "Formatted at 2026-06-21 06:53:10 on dist-test-slave-gmf2"
server_key: "fe035e01470e8e2dde790d3175f1e235"
server_key_iv: "0e01bf2c420f169c0ab6b112e9197953"
server_key_version: "encryptionkey@0"
I20260621 06:53:10.246189 30950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "c3b953cbf08a4fb8b113bd8ccff060a4"
format_stamp: "Formatted at 2026-06-21 06:53:10 on dist-test-slave-gmf2"
server_key: "fe035e01470e8e2dde790d3175f1e235"
server_key_iv: "0e01bf2c420f169c0ab6b112e9197953"
server_key_version: "encryptionkey@0"
I20260621 06:53:10.251204 30950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260621 06:53:10.254743 30964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:10.256059 30950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:53:10.256237 30950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "c3b953cbf08a4fb8b113bd8ccff060a4"
format_stamp: "Formatted at 2026-06-21 06:53:10 on dist-test-slave-gmf2"
server_key: "fe035e01470e8e2dde790d3175f1e235"
server_key_iv: "0e01bf2c420f169c0ab6b112e9197953"
server_key_version: "encryptionkey@0"
I20260621 06:53:10.256459 30950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:10.262223 30950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:10.263000 30950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:10.263279 30950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:10.264189 30950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:53:10.265558 30950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:53:10.265695 30950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:10.265836 30950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:53:10.265951 30950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:10.299096 30950 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:36815
I20260621 06:53:10.299180 31076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:36815 every 8 connection(s)
I20260621 06:53:10.300398 30950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:53:10.308491 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:10.303028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:32833 (local address 127.20.120.254:35639)
0621 06:53:10.303213 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:10.303218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:10.304708 (+ 1490us) server_negotiation.cc:407] Connection header received
0621 06:53:10.306482 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:10.306486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:10.306546 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:10.306634 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:10.307879 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:10.307882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:10.307885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:10.307899 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:10.307927 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:10.307938 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:10.308026 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:10.308244 (+ 218us) server_negotiation.cc:299] Negotiation successful
0621 06:53:10.308290 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 06:53:10.308885 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 30950
I20260621 06:53:10.309072 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:53:10.309465 20963 external_mini_cluster.cc:1428] Setting key d429742b6d24a407f453271b5fdbc81f
I20260621 06:53:10.309530 31079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:10.303570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:35639 (local address 127.20.120.194:32833)
0621 06:53:10.304526 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:10.304571 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:10.306218 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:10.306852 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:10.306860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:10.306945 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:10.307597 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:53:10.307600 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:10.307635 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:53:10.307729 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:53:10.308022 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:53:10.308030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:53:10.308035 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 06:53:10.308283 (+ 248us) client_negotiation.cc:240] Negotiation successful
0621 06:53:10.308481 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":282,"threads_started":1}
I20260621 06:53:10.310973 31077 heartbeater.cc:347] Connected to a master server at 127.20.120.254:35639
I20260621 06:53:10.311247 31077 heartbeater.cc:464] Registering TS with master...
I20260621 06:53:10.311863 31077 heartbeater.cc:511] Master 127.20.120.254:35639 requested a full tablet report, sending...
I20260621 06:53:10.312740 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:35639
--builtin_ntp_servers=127.20.120.212:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 06:53:10.313329 30759 ts_manager.cc:194] Registered new tserver with Master: c3b953cbf08a4fb8b113bd8ccff060a4 (127.20.120.194:36815)
W20260621 06:53:10.637712 31080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:10.638005 31080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:10.638135 31080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:10.645258 31080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:10.645349 31080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:10.645483 31080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:53:10.654364 31080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:35639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:10.655694 31080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:10.657498 31080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:10.670888 31085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:10.670912 31086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:10.671154 31088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:10.671473 31080 server_base.cc:1034] running on GCE node
I20260621 06:53:10.672097 31080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:10.672992 31080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:10.674189 31080 hybrid_clock.cc:648] HybridClock initialized: now 1782024790674139 us; error 45 us; skew 500 ppm
I20260621 06:53:10.674506 31080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:53:10.676638 31080 webserver.cc:460] Webserver started at http://127.20.120.195:43875/ using document root <none> and password file <none>
I20260621 06:53:10.677358 31080 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:10.677471 31080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:10.677789 31080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:53:10.680564 31080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "a4bfe1c87f0c444fbfa5b68b2600842e"
format_stamp: "Formatted at 2026-06-21 06:53:10 on dist-test-slave-gmf2"
server_key: "def42309f62fe78a494385bf86f9f72f"
server_key_iv: "b78f0853e6f6dd504ce572c1a525da63"
server_key_version: "encryptionkey@0"
I20260621 06:53:10.681327 31080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "a4bfe1c87f0c444fbfa5b68b2600842e"
format_stamp: "Formatted at 2026-06-21 06:53:10 on dist-test-slave-gmf2"
server_key: "def42309f62fe78a494385bf86f9f72f"
server_key_iv: "b78f0853e6f6dd504ce572c1a525da63"
server_key_version: "encryptionkey@0"
I20260621 06:53:10.686327 31080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 06:53:10.689556 31094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:10.690897 31080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:53:10.691112 31080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "a4bfe1c87f0c444fbfa5b68b2600842e"
format_stamp: "Formatted at 2026-06-21 06:53:10 on dist-test-slave-gmf2"
server_key: "def42309f62fe78a494385bf86f9f72f"
server_key_iv: "b78f0853e6f6dd504ce572c1a525da63"
server_key_version: "encryptionkey@0"
I20260621 06:53:10.691310 31080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:10.696398 31080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:10.697268 31080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:10.697543 31080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:10.698478 31080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:53:10.699760 31080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:53:10.699873 31080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:10.699992 31080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:53:10.700090 31080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:10.732527 31080 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:36565
I20260621 06:53:10.732573 31206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:36565 every 8 connection(s)
I20260621 06:53:10.733860 31080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:53:10.734921 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 31080
I20260621 06:53:10.735116 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:53:10.735661 20963 external_mini_cluster.cc:1428] Setting key f4de0923dc05cda06369af95acd3dd05
I20260621 06:53:10.742637 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:10.736719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:60671 (local address 127.20.120.254:35639)
0621 06:53:10.736909 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:10.736915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:10.738732 (+ 1817us) server_negotiation.cc:407] Connection header received
0621 06:53:10.740418 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:10.740423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:10.740487 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:10.740602 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:10.741884 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:10.741890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:10.741895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:10.741934 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:10.741969 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:10.741983 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:10.742133 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:10.742318 (+ 185us) server_negotiation.cc:299] Negotiation successful
0621 06:53:10.742380 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 06:53:10.743471 31209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:10.737407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:35639 (local address 127.20.120.195:60671)
0621 06:53:10.738562 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:10.738621 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:10.740165 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:10.740827 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:10.740835 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:10.740919 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:10.741600 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:53:10.741602 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:10.741638 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:53:10.741724 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:53:10.742089 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:53:10.742100 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:53:10.742109 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 06:53:10.742397 (+ 288us) client_negotiation.cc:240] Negotiation successful
0621 06:53:10.742627 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":368,"threads_started":1}
I20260621 06:53:10.744731 31207 heartbeater.cc:347] Connected to a master server at 127.20.120.254:35639
I20260621 06:53:10.744997 31207 heartbeater.cc:464] Registering TS with master...
I20260621 06:53:10.745505 31207 heartbeater.cc:511] Master 127.20.120.254:35639 requested a full tablet report, sending...
I20260621 06:53:10.746860 30759 ts_manager.cc:194] Registered new tserver with Master: a4bfe1c87f0c444fbfa5b68b2600842e (127.20.120.195:36565)
I20260621 06:53:10.751070 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 06:53:10.757027 30948 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:53:10.755442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.20.120.254:35639)
0621 06:53:10.755590 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:10.755594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:10.756213 (+ 619us) server_negotiation.cc:407] Connection header received
0621 06:53:10.756343 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:10.756346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:10.756395 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:10.756504 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:10.756788 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37684: BlockingRecv error: recv got EOF from 127.0.0.1:37684 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 06:53:10.757395 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 30819
I20260621 06:53:10.760751 31216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:10.758436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.20.120.193:45727)
0621 06:53:10.758910 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:10.758916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:10.758939 (+ 23us) server_negotiation.cc:407] Connection header received
0621 06:53:10.758990 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:10.758995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:10.759159 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:10.759270 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:10.760193 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:10.760198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:10.760208 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:10.760220 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:10.760259 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:10.760276 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:10.760368 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:10.760425 (+ 57us) server_negotiation.cc:299] Negotiation successful
0621 06:53:10.760494 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":226,"threads_started":1}
I20260621 06:53:10.761819 30941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:10.854869 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 30819
I20260621 06:53:10.874730 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 30950
I20260621 06:53:10.878507 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:10.875728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.20.120.194:36815)
0621 06:53:10.876295 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:10.876302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:10.876332 (+ 30us) server_negotiation.cc:407] Connection header received
0621 06:53:10.876399 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:10.876406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:10.876658 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:10.876800 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:10.877703 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:10.877710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:10.877720 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:10.877742 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:10.877788 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:10.877803 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:10.877969 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:10.878053 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 06:53:10.878187 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
I20260621 06:53:10.879710 31072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:10.960211 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 30950
I20260621 06:53:10.976660 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 31080
I20260621 06:53:10.980909 31220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:10.977625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.20.120.195:36565)
0621 06:53:10.978334 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:10.978341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:10.978372 (+ 31us) server_negotiation.cc:407] Connection header received
0621 06:53:10.978439 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:10.978446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:10.978679 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:10.978861 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:10.980112 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:10.980119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:10.980132 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:53:10.980149 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:10.980191 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:53:10.980206 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:10.980404 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:10.980497 (+ 93us) server_negotiation.cc:299] Negotiation successful
0621 06:53:10.980603 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":396,"threads_started":1}
I20260621 06:53:10.982219 31202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:11.062067 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 31080
I20260621 06:53:11.077543 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 30729
I20260621 06:53:11.078763 30790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:11.179000 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 30729
2026-06-21T06:53:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2307 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:53:11 dist-test-slave-gmf2 krb5kdc[31225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:53:11 dist-test-slave-gmf2 krb5kdc[31225](info): set up 2 sockets
Jun 21 06:53:11 dist-test-slave-gmf2 krb5kdc[31225](info): commencing operation
krb5kdc: starting...
W20260621 06:53:12.266839 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:53:12 dist-test-slave-gmf2 krb5kdc[31225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:53:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:53:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:53:12.478780 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34091
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:34091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:53:12.790405 31238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:12.790699 31238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:12.790860 31238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:12.797798 31238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:53:12.797981 31238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:12.798082 31238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:12.798182 31238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:53:12.798265 31238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:53:12.807668 31238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:34091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:12.809006 31238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:12.810329 31238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:12.821532 31243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:12.821570 31244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:12.821637 31246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:12.822642 31238 server_base.cc:1034] running on GCE node
I20260621 06:53:12.823351 31238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:12.824633 31238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:12.825856 31238 hybrid_clock.cc:648] HybridClock initialized: now 1782024792825811 us; error 54 us; skew 500 ppm
I20260621 06:53:12.826190 31238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:53:12 dist-test-slave-gmf2 krb5kdc[31225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024792, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:53:12.829267 31238 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:53:12.830648 31238 webserver.cc:460] Webserver started at http://127.20.120.254:43063/ using document root <none> and password file <none>
I20260621 06:53:12.831274 31238 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:12.831413 31238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:12.831697 31238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:53:12.834643 31238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "af12b4e9f6a0478fb5154e7778f69e9e"
format_stamp: "Formatted at 2026-06-21 06:53:12 on dist-test-slave-gmf2"
server_key: "f9ac2aed8f624e378367e6c65b950ea6"
server_key_iv: "f1af260dd552347684bacf80d1453ef1"
server_key_version: "encryptionkey@0"
I20260621 06:53:12.835346 31238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "af12b4e9f6a0478fb5154e7778f69e9e"
format_stamp: "Formatted at 2026-06-21 06:53:12 on dist-test-slave-gmf2"
server_key: "f9ac2aed8f624e378367e6c65b950ea6"
server_key_iv: "f1af260dd552347684bacf80d1453ef1"
server_key_version: "encryptionkey@0"
I20260621 06:53:12.840062 31238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 06:53:12.843257 31253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:12.844475 31238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 06:53:12.844651 31238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "af12b4e9f6a0478fb5154e7778f69e9e"
format_stamp: "Formatted at 2026-06-21 06:53:12 on dist-test-slave-gmf2"
server_key: "f9ac2aed8f624e378367e6c65b950ea6"
server_key_iv: "f1af260dd552347684bacf80d1453ef1"
server_key_version: "encryptionkey@0"
I20260621 06:53:12.844822 31238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:12.871265 31238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:12.875576 31238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:12.875850 31238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:12.893736 31238 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:34091
I20260621 06:53:12.893777 31304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:34091 every 8 connection(s)
I20260621 06:53:12.895027 31238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:53:12.897832 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 31238
I20260621 06:53:12.898064 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:53:12.898661 31305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:53:12.898666 20963 external_mini_cluster.cc:1428] Setting key d38600c7a548641da94dccec71bf248c
Jun 21 06:53:12 dist-test-slave-gmf2 krb5kdc[31225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:53:12.913553 31305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e: Bootstrap starting.
I20260621 06:53:12.916908 31305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e: Neither blocks nor log segments found. Creating new log.
I20260621 06:53:12.918113 31305 log.cc:824] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e: Log is configured to *not* fsync() on all Append() calls
I20260621 06:53:12.921172 31305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e: No bootstrap required, opened a new log
I20260621 06:53:12.922045 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:12.900552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.20.120.254:34091)
0621 06:53:12.901354 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:12.901365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:12.901409 (+ 44us) server_negotiation.cc:407] Connection header received
0621 06:53:12.903138 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:12.903158 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:12.903537 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:12.903866 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:12.906005 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:12.907187 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:12.908950 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:12.909553 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:12.909713 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:12.914749 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:12.914758 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:12.914772 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:12.914810 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:12.917860 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:12.918657 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:12.918668 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:12.918680 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:12.918845 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:12.919235 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:12.919245 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:12.919252 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:12.919774 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:12.920074 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:12.920489 (+ 415us) server_negotiation.cc:299] Negotiation successful
0621 06:53:12.920906 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":371,"threads_started":1}
I20260621 06:53:12.929796 31305 raft_consensus.cc:348] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af12b4e9f6a0478fb5154e7778f69e9e" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34091 } }
I20260621 06:53:12.930255 31305 raft_consensus.cc:374] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:53:12.930366 31305 raft_consensus.cc:729] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af12b4e9f6a0478fb5154e7778f69e9e, State: Initialized, Role: FOLLOWER
I20260621 06:53:12.930928 31305 consensus_queue.cc:260] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af12b4e9f6a0478fb5154e7778f69e9e" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34091 } }
I20260621 06:53:12.931173 31305 raft_consensus.cc:388] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:53:12.931313 31305 raft_consensus.cc:482] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:53:12.931504 31305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:53:12.933823 31305 raft_consensus.cc:504] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af12b4e9f6a0478fb5154e7778f69e9e" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34091 } }
I20260621 06:53:12.934466 31305 leader_election.cc:302] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af12b4e9f6a0478fb5154e7778f69e9e; no voters:
I20260621 06:53:12.935254 31305 leader_election.cc:288] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:53:12.935395 31310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:53:12.936197 31310 raft_consensus.cc:686] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [term 1 LEADER]: Becoming Leader. State: Replica: af12b4e9f6a0478fb5154e7778f69e9e, State: Running, Role: LEADER
I20260621 06:53:12.936802 31310 consensus_queue.cc:237] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af12b4e9f6a0478fb5154e7778f69e9e" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34091 } }
I20260621 06:53:12.937629 31305 sys_catalog.cc:564] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:53:12.939798 31311 sys_catalog.cc:455] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af12b4e9f6a0478fb5154e7778f69e9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af12b4e9f6a0478fb5154e7778f69e9e" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34091 } } }
I20260621 06:53:12.939788 31312 sys_catalog.cc:455] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader af12b4e9f6a0478fb5154e7778f69e9e. Latest consensus state: current_term: 1 leader_uuid: "af12b4e9f6a0478fb5154e7778f69e9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af12b4e9f6a0478fb5154e7778f69e9e" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34091 } } }
I20260621 06:53:12.940263 31312 sys_catalog.cc:458] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [sys.catalog]: This master's current role is: LEADER
I20260621 06:53:12.940156 31311 sys_catalog.cc:458] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e [sys.catalog]: This master's current role is: LEADER
I20260621 06:53:12.941461 31317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:53:12.945377 31317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:53:12.954843 31317 catalog_manager.cc:1347] Generated new cluster ID: cacaf014e01748efa6aff2b3410d67cc
I20260621 06:53:12.954981 31317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:53:12.973132 31317 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:53:12.974502 31317 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:53:12.986678 31317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e: Generated new TSK 0
I20260621 06:53:12.987717 31317 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:53:13.053187 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34091
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:53:13.389410 31331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:13.389653 31331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:13.389743 31331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:13.396826 31331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:13.396931 31331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:13.397044 31331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:53:13.406363 31331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:13.407512 31331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:13.408746 31331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:13.422022 31337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:13.422286 31336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:13.421991 31339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:13.422919 31331 server_base.cc:1034] running on GCE node
I20260621 06:53:13.423583 31331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:13.424537 31331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:13.425767 31331 hybrid_clock.cc:648] HybridClock initialized: now 1782024793425724 us; error 57 us; skew 500 ppm
I20260621 06:53:13.426095 31331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:53:13 dist-test-slave-gmf2 krb5kdc[31225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024793, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:53:13.432186 31331 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:53:13.433481 31331 webserver.cc:460] Webserver started at http://127.20.120.193:37385/ using document root <none> and password file <none>
I20260621 06:53:13.434177 31331 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:13.434273 31331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:13.434500 31331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:53:13.438862 31331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "2d27db21ae2c4152b584ef4ac372d9e8"
format_stamp: "Formatted at 2026-06-21 06:53:13 on dist-test-slave-gmf2"
server_key: "8f2fb80ac84cb8db9a5db3c0055d5a75"
server_key_iv: "0c0a1f321d44bef6c7b8fcafe3c124d8"
server_key_version: "encryptionkey@0"
I20260621 06:53:13.439821 31331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "2d27db21ae2c4152b584ef4ac372d9e8"
format_stamp: "Formatted at 2026-06-21 06:53:13 on dist-test-slave-gmf2"
server_key: "8f2fb80ac84cb8db9a5db3c0055d5a75"
server_key_iv: "0c0a1f321d44bef6c7b8fcafe3c124d8"
server_key_version: "encryptionkey@0"
I20260621 06:53:13.445540 31331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 06:53:13.449128 31346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:13.450484 31331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 06:53:13.450655 31331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "2d27db21ae2c4152b584ef4ac372d9e8"
format_stamp: "Formatted at 2026-06-21 06:53:13 on dist-test-slave-gmf2"
server_key: "8f2fb80ac84cb8db9a5db3c0055d5a75"
server_key_iv: "0c0a1f321d44bef6c7b8fcafe3c124d8"
server_key_version: "encryptionkey@0"
I20260621 06:53:13.450829 31331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:13.466476 31331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:13.470511 31331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:13.470799 31331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:13.471662 31331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:53:13.472913 31331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:53:13.473011 31331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:13.473155 31331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:53:13.473227 31331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:13.504122 31331 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:36505
I20260621 06:53:13.504154 31458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:36505 every 8 connection(s)
I20260621 06:53:13.505344 31331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:53:13.510069 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 31331
I20260621 06:53:13.510277 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:53:13.510880 20963 external_mini_cluster.cc:1428] Setting key a5059220e26692f1b07799ea2f77705f
Jun 21 06:53:13 dist-test-slave-gmf2 krb5kdc[31225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024793, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:53:13.527266 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:13.507866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:47393 (local address 127.20.120.254:34091)
0621 06:53:13.508294 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:13.508298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:13.509110 (+ 812us) server_negotiation.cc:407] Connection header received
0621 06:53:13.511067 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:13.511072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:13.511153 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:13.511283 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:13.512795 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:13.513487 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:13.515975 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:13.516339 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:13.516424 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:13.520657 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:13.520666 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:13.520672 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:13.520713 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:13.524182 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:13.524826 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:13.524834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:13.524840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:13.524987 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:13.525360 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:13.525368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:13.525374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:13.525724 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:13.525863 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:13.526576 (+ 713us) server_negotiation.cc:299] Negotiation successful
0621 06:53:13.526804 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":236,"threads_started":1}
I20260621 06:53:13.527839 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:13.508299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34091 (local address 127.20.120.193:47393)
0621 06:53:13.508953 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:13.508995 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:13.510782 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:13.511479 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:13.511487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:13.511975 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:13.512644 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:13.512662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:13.513640 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:13.513648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:13.515832 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:13.515847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:13.516550 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:13.516558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:13.516850 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:13.517513 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:13.517556 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:13.520409 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:53:13.524307 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:53:13.524313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:53:13.524324 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:53:13.524694 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:53:13.525095 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:53:13.525098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:53:13.525100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:53:13.525248 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:53:13.525832 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:53:13.525837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:53:13.526197 (+ 360us) client_negotiation.cc:769] Sending connection context
0621 06:53:13.526415 (+ 218us) client_negotiation.cc:240] Negotiation successful
0621 06:53:13.526748 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":274,"threads_started":1}
I20260621 06:53:13.529235 31459 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34091
I20260621 06:53:13.529510 31459 heartbeater.cc:464] Registering TS with master...
I20260621 06:53:13.530243 31459 heartbeater.cc:511] Master 127.20.120.254:34091 requested a full tablet report, sending...
I20260621 06:53:13.532166 31270 ts_manager.cc:194] Registered new tserver with Master: 2d27db21ae2c4152b584ef4ac372d9e8 (127.20.120.193:36505)
I20260621 06:53:13.533531 31270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:47393
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:53:13.561532 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34091
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:53:13.876247 31465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:13.876504 31465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:13.876662 31465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:13.883528 31465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:13.883653 31465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:13.883774 31465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:53:13.893044 31465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:13.894312 31465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:13.895527 31465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:13.908568 31471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:13.908540 31473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:13.908794 31470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:13.909425 31465 server_base.cc:1034] running on GCE node
I20260621 06:53:13.910104 31465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:13.911016 31465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:13.912273 31465 hybrid_clock.cc:648] HybridClock initialized: now 1782024793912194 us; error 53 us; skew 500 ppm
I20260621 06:53:13.912648 31465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:53:13 dist-test-slave-gmf2 krb5kdc[31225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024793, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:53:13.915874 31465 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:53:13.917320 31465 webserver.cc:460] Webserver started at http://127.20.120.194:43851/ using document root <none> and password file <none>
I20260621 06:53:13.918113 31465 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:13.918269 31465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:13.918591 31465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:53:13.921458 31465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "bbebee7b12a949d2b3b6ac148993faad"
format_stamp: "Formatted at 2026-06-21 06:53:13 on dist-test-slave-gmf2"
server_key: "b807642c25018d87e2978b2767ca12d4"
server_key_iv: "147c452aa933c8de1c86a91c4cd12415"
server_key_version: "encryptionkey@0"
I20260621 06:53:13.922331 31465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "bbebee7b12a949d2b3b6ac148993faad"
format_stamp: "Formatted at 2026-06-21 06:53:13 on dist-test-slave-gmf2"
server_key: "b807642c25018d87e2978b2767ca12d4"
server_key_iv: "147c452aa933c8de1c86a91c4cd12415"
server_key_version: "encryptionkey@0"
I20260621 06:53:13.927554 31465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 06:53:13.930989 31480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:13.932289 31465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:53:13.932482 31465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "bbebee7b12a949d2b3b6ac148993faad"
format_stamp: "Formatted at 2026-06-21 06:53:13 on dist-test-slave-gmf2"
server_key: "b807642c25018d87e2978b2767ca12d4"
server_key_iv: "147c452aa933c8de1c86a91c4cd12415"
server_key_version: "encryptionkey@0"
I20260621 06:53:13.932686 31465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:13.957082 31465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:13.961128 31465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:13.961421 31465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:13.962428 31465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:53:13.963725 31465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:53:13.963826 31465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:13.963972 31465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:53:13.964044 31465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:13.996322 31465 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:36377
I20260621 06:53:13.996371 31592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:36377 every 8 connection(s)
I20260621 06:53:13.997596 31465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:53:14.000569 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 31465
I20260621 06:53:14.000770 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:53:14.001343 20963 external_mini_cluster.cc:1428] Setting key 922d4e060f2ba7adc8bda10d4de038fe
Jun 21 06:53:14 dist-test-slave-gmf2 krb5kdc[31225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024793, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:53:14.018996 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.000184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:39303 (local address 127.20.120.254:34091)
0621 06:53:14.000368 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:14.000373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:14.001859 (+ 1486us) server_negotiation.cc:407] Connection header received
0621 06:53:14.004087 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:14.004092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:14.004175 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:14.004277 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:14.005728 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.006489 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.007985 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.008400 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.008528 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.011953 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:14.011961 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:14.011969 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:14.012011 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:14.015590 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:14.016250 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:14.016259 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:14.016265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:14.016410 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:14.016941 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:14.016948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:14.016954 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:14.017344 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:14.017519 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:14.018293 (+ 774us) server_negotiation.cc:299] Negotiation successful
0621 06:53:14.018540 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 06:53:14.019645 31595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.000910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34091 (local address 127.20.120.194:39303)
0621 06:53:14.001693 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:14.001745 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:14.003828 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:14.004518 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:14.004526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:14.005004 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:14.005569 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.005585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.006644 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.006648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.007851 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.007861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.008608 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.008615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.008838 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.009481 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:14.009502 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:14.011755 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:53:14.015721 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:53:14.015728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:53:14.015739 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:53:14.016127 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:53:14.016569 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:53:14.016576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:53:14.016582 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:53:14.016821 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:53:14.017471 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:53:14.017495 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:53:14.017838 (+ 343us) client_negotiation.cc:769] Sending connection context
0621 06:53:14.018136 (+ 298us) client_negotiation.cc:240] Negotiation successful
0621 06:53:14.018508 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":311,"threads_started":1}
I20260621 06:53:14.021085 31593 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34091
I20260621 06:53:14.021427 31593 heartbeater.cc:464] Registering TS with master...
I20260621 06:53:14.022351 31593 heartbeater.cc:511] Master 127.20.120.254:34091 requested a full tablet report, sending...
I20260621 06:53:14.023936 31270 ts_manager.cc:194] Registered new tserver with Master: bbebee7b12a949d2b3b6ac148993faad (127.20.120.194:36377)
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
I20260621 06:53:14.024776 31270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:39303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:53:14.051080 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34091
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:53:14.360158 31598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:14.360436 31598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:14.360589 31598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:14.367362 31598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:14.367451 31598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:14.367656 31598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:53:14.376760 31598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:14.377971 31598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:14.379189 31598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:14.392096 31606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:14.392186 31603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:14.392283 31604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:14.392892 31598 server_base.cc:1034] running on GCE node
I20260621 06:53:14.393505 31598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:14.394445 31598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:14.395637 31598 hybrid_clock.cc:648] HybridClock initialized: now 1782024794395578 us; error 52 us; skew 500 ppm
I20260621 06:53:14.395925 31598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:53:14 dist-test-slave-gmf2 krb5kdc[31225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024794, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:53:14.398681 31598 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:53:14.399969 31598 webserver.cc:460] Webserver started at http://127.20.120.195:38497/ using document root <none> and password file <none>
I20260621 06:53:14.400606 31598 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:14.400743 31598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:14.401038 31598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:53:14.403772 31598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "cdb716b66b8d4215a546fb07a413fd30"
format_stamp: "Formatted at 2026-06-21 06:53:14 on dist-test-slave-gmf2"
server_key: "f92ff3ad2148ccf56d767a3923b3b23d"
server_key_iv: "33ba97b2ecce8c1a2bab71e76a1257b1"
server_key_version: "encryptionkey@0"
I20260621 06:53:14.404516 31598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "cdb716b66b8d4215a546fb07a413fd30"
format_stamp: "Formatted at 2026-06-21 06:53:14 on dist-test-slave-gmf2"
server_key: "f92ff3ad2148ccf56d767a3923b3b23d"
server_key_iv: "33ba97b2ecce8c1a2bab71e76a1257b1"
server_key_version: "encryptionkey@0"
I20260621 06:53:14.409183 31598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260621 06:53:14.412364 31613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:14.413645 31598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:53:14.413803 31598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "cdb716b66b8d4215a546fb07a413fd30"
format_stamp: "Formatted at 2026-06-21 06:53:14 on dist-test-slave-gmf2"
server_key: "f92ff3ad2148ccf56d767a3923b3b23d"
server_key_iv: "33ba97b2ecce8c1a2bab71e76a1257b1"
server_key_version: "encryptionkey@0"
I20260621 06:53:14.414047 31598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:14.434577 31598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:14.438648 31598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:14.438925 31598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:14.439795 31598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:53:14.440990 31598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:53:14.441114 31598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:14.441247 31598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:53:14.441313 31598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:14.473290 31598 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:41233
I20260621 06:53:14.473340 31725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:41233 every 8 connection(s)
I20260621 06:53:14.474550 31598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:53:14.479370 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 31598
I20260621 06:53:14.479573 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:53:14.479987 20963 external_mini_cluster.cc:1428] Setting key d305d9870b62e6df475c501309999817
Jun 21 06:53:14 dist-test-slave-gmf2 krb5kdc[31225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024794, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:53:14.493553 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.477102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:50669 (local address 127.20.120.254:34091)
0621 06:53:14.477282 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:14.477286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:14.478487 (+ 1201us) server_negotiation.cc:407] Connection header received
0621 06:53:14.480426 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:14.480432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:14.480517 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:14.480632 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:14.482094 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.482983 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.484456 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.484877 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.485005 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.488128 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:14.488133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:14.488136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:14.488171 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:14.490704 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:14.491357 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:14.491368 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:14.491376 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:14.491493 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:14.491881 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:14.491886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:14.491888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:14.492131 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:14.492307 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:14.492930 (+ 623us) server_negotiation.cc:299] Negotiation successful
0621 06:53:14.493141 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 06:53:14.494679 31728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.477561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34091 (local address 127.20.120.195:50669)
0621 06:53:14.478331 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:14.478368 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:14.480155 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:14.480861 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:14.480869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:14.481347 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:14.481883 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.481899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.483133 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.483139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.484317 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.484328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.485083 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.485093 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.485328 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.485946 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:14.485978 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:14.487936 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:53:14.490841 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:53:14.490849 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:53:14.490863 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:53:14.491240 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:53:14.491601 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:53:14.491606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:53:14.491610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:53:14.491768 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:53:14.492261 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:53:14.492268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:53:14.492602 (+ 334us) client_negotiation.cc:769] Sending connection context
0621 06:53:14.492844 (+ 242us) client_negotiation.cc:240] Negotiation successful
0621 06:53:14.493310 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":343,"threads_started":1}
I20260621 06:53:14.495904 31726 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34091
I20260621 06:53:14.496187 31726 heartbeater.cc:464] Registering TS with master...
I20260621 06:53:14.496826 31726 heartbeater.cc:511] Master 127.20.120.254:34091 requested a full tablet report, sending...
I20260621 06:53:14.498086 31270 ts_manager.cc:194] Registered new tserver with Master: cdb716b66b8d4215a546fb07a413fd30 (127.20.120.195:41233)
I20260621 06:53:14.498867 31270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:50669
I20260621 06:53:14.506880 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:53:14.523756 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.511697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.20.120.254:34091)
0621 06:53:14.511916 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:14.511925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:14.512260 (+ 335us) server_negotiation.cc:407] Connection header received
0621 06:53:14.512404 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:14.512408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:14.512465 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:14.512541 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:14.513840 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.514574 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.515731 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.516081 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.516162 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.518732 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:14.518736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:14.518739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:14.518767 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:14.521190 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:14.521821 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:14.521825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:14.521827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:14.521999 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:14.522405 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:14.522409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:14.522411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:14.522692 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:14.522866 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:14.523157 (+ 291us) server_negotiation.cc:299] Negotiation successful
0621 06:53:14.523294 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 06:53:14.527374 31270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:53:14.533667 31270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:53:14.536157 31459 heartbeater.cc:503] Master 127.20.120.254:34091 was elected leader, sending a full tablet report...
I20260621 06:53:14.557066 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.545638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44492 (local address 127.20.120.194:36377)
0621 06:53:14.546931 (+ 1293us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:14.546939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:14.546964 (+ 25us) server_negotiation.cc:407] Connection header received
0621 06:53:14.547052 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:14.547062 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:14.547340 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:14.548365 (+ 1025us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:53:14.548646 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.550627 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.554891 (+ 4264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.555987 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.556162 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.556322 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:14.556608 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 06:53:14.556765 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1144,"thread_start_us":1006,"threads_started":1}
I20260621 06:53:14.557102 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.545157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:36377 (local address 127.0.0.1:44492)
0621 06:53:14.546350 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:14.546366 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:14.546489 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:14.547644 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:14.547654 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:14.547691 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:53:14.548156 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.548171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.552062 (+ 3891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.552067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.554715 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.554728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.556171 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.556176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.556359 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.556421 (+ 62us) client_negotiation.cc:769] Sending connection context
0621 06:53:14.556551 (+ 130us) client_negotiation.cc:240] Negotiation successful
0621 06:53:14.556716 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"thread_start_us":959,"threads_started":1}
I20260621 06:53:14.558809 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.544796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:36505 (local address 127.0.0.1:52146)
0621 06:53:14.545627 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:14.545660 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:14.545815 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:14.551841 (+ 6026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:14.551846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:14.551872 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:53:14.552367 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.552381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.554487 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.554493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.556178 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.556190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.557927 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.557934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.558164 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.558197 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 06:53:14.558296 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 06:53:14.558421 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":39,"thread_start_us":456,"threads_started":1}
I20260621 06:53:14.558815 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.544925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.20.120.193:36505)
0621 06:53:14.551315 (+ 6390us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:14.551323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:14.551346 (+ 23us) server_negotiation.cc:407] Connection header received
0621 06:53:14.551426 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:14.551433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:14.551653 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:14.552573 (+ 920us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:53:14.552794 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.554353 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.556322 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.557703 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.557886 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.558053 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:14.558349 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 06:53:14.558505 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6184,"thread_start_us":5995,"threads_started":1}
I20260621 06:53:14.560968 31528 tablet_service.cc:1467] Processing CreateTablet for tablet 7683a11ee19841a0a1fbdde7be11897c (DEFAULT_TABLE table=test-table [id=3ca877a81d6844dfaea429790fb6b65b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:53:14.562224 31528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7683a11ee19841a0a1fbdde7be11897c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:53:14.561949 31394 tablet_service.cc:1467] Processing CreateTablet for tablet 7683a11ee19841a0a1fbdde7be11897c (DEFAULT_TABLE table=test-table [id=3ca877a81d6844dfaea429790fb6b65b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:53:14.563519 31394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7683a11ee19841a0a1fbdde7be11897c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:53:14.563982 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.546961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:41233 (local address 127.0.0.1:56564)
0621 06:53:14.551595 (+ 4634us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:14.551616 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:14.551753 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:14.552809 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:14.552813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:14.552836 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:53:14.553248 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.553256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.554766 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.554773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.558232 (+ 3459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.558243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.563164 (+ 4921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.563170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.563342 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.563374 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 06:53:14.563470 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 06:53:14.563595 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4469,"mutex_wait_us":46,"thread_start_us":4320,"threads_started":1}
I20260621 06:53:14.564679 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.547353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.20.120.195:41233)
0621 06:53:14.552340 (+ 4987us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:14.552345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:14.552368 (+ 23us) server_negotiation.cc:407] Connection header received
0621 06:53:14.552430 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:14.552436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:14.552642 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:14.552817 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:53:14.553418 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.554602 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.561557 (+ 6955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.562959 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.563150 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.563299 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:14.564198 (+ 899us) server_negotiation.cc:299] Negotiation successful
0621 06:53:14.564352 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4834,"thread_start_us":4673,"threads_started":1}
I20260621 06:53:14.568075 31661 tablet_service.cc:1467] Processing CreateTablet for tablet 7683a11ee19841a0a1fbdde7be11897c (DEFAULT_TABLE table=test-table [id=3ca877a81d6844dfaea429790fb6b65b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:53:14.569643 31661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7683a11ee19841a0a1fbdde7be11897c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:53:14.573472 31744 tablet_bootstrap.cc:492] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad: Bootstrap starting.
I20260621 06:53:14.576591 31744 tablet_bootstrap.cc:654] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad: Neither blocks nor log segments found. Creating new log.
I20260621 06:53:14.577886 31744 log.cc:824] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad: Log is configured to *not* fsync() on all Append() calls
I20260621 06:53:14.578163 31745 tablet_bootstrap.cc:492] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8: Bootstrap starting.
I20260621 06:53:14.581283 31744 tablet_bootstrap.cc:492] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad: No bootstrap required, opened a new log
I20260621 06:53:14.581458 31745 tablet_bootstrap.cc:654] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8: Neither blocks nor log segments found. Creating new log.
I20260621 06:53:14.581535 31744 ts_tablet_manager.cc:1397] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 06:53:14.582629 31745 log.cc:824] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8: Log is configured to *not* fsync() on all Append() calls
I20260621 06:53:14.582957 31747 tablet_bootstrap.cc:492] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30: Bootstrap starting.
I20260621 06:53:14.586051 31747 tablet_bootstrap.cc:654] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30: Neither blocks nor log segments found. Creating new log.
I20260621 06:53:14.586148 31745 tablet_bootstrap.cc:492] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8: No bootstrap required, opened a new log
I20260621 06:53:14.586431 31745 ts_tablet_manager.cc:1397] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 06:53:14.587294 31747 log.cc:824] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30: Log is configured to *not* fsync() on all Append() calls
I20260621 06:53:14.589910 31747 tablet_bootstrap.cc:492] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30: No bootstrap required, opened a new log
I20260621 06:53:14.590226 31747 ts_tablet_manager.cc:1397] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260621 06:53:14.590330 31744 raft_consensus.cc:348] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d27db21ae2c4152b584ef4ac372d9e8" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36505 } } peers { permanent_uuid: "bbebee7b12a949d2b3b6ac148993faad" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36377 } } peers { permanent_uuid: "cdb716b66b8d4215a546fb07a413fd30" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41233 } }
I20260621 06:53:14.590710 31744 raft_consensus.cc:374] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:53:14.590835 31744 raft_consensus.cc:729] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbebee7b12a949d2b3b6ac148993faad, State: Initialized, Role: FOLLOWER
I20260621 06:53:14.591398 31744 consensus_queue.cc:260] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d27db21ae2c4152b584ef4ac372d9e8" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36505 } } peers { permanent_uuid: "bbebee7b12a949d2b3b6ac148993faad" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36377 } } peers { permanent_uuid: "cdb716b66b8d4215a546fb07a413fd30" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41233 } }
I20260621 06:53:14.592391 31593 heartbeater.cc:503] Master 127.20.120.254:34091 was elected leader, sending a full tablet report...
I20260621 06:53:14.593093 31744 ts_tablet_manager.cc:1428] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260621 06:53:14.596839 31745 raft_consensus.cc:348] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d27db21ae2c4152b584ef4ac372d9e8" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36505 } } peers { permanent_uuid: "bbebee7b12a949d2b3b6ac148993faad" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36377 } } peers { permanent_uuid: "cdb716b66b8d4215a546fb07a413fd30" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41233 } }
I20260621 06:53:14.597234 31745 raft_consensus.cc:374] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:53:14.597335 31745 raft_consensus.cc:729] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d27db21ae2c4152b584ef4ac372d9e8, State: Initialized, Role: FOLLOWER
I20260621 06:53:14.597961 31745 consensus_queue.cc:260] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d27db21ae2c4152b584ef4ac372d9e8" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36505 } } peers { permanent_uuid: "bbebee7b12a949d2b3b6ac148993faad" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36377 } } peers { permanent_uuid: "cdb716b66b8d4215a546fb07a413fd30" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41233 } }
I20260621 06:53:14.599545 31745 ts_tablet_manager.cc:1428] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260621 06:53:14.600083 31747 raft_consensus.cc:348] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d27db21ae2c4152b584ef4ac372d9e8" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36505 } } peers { permanent_uuid: "bbebee7b12a949d2b3b6ac148993faad" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36377 } } peers { permanent_uuid: "cdb716b66b8d4215a546fb07a413fd30" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41233 } }
I20260621 06:53:14.600492 31747 raft_consensus.cc:374] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:53:14.600611 31747 raft_consensus.cc:729] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb716b66b8d4215a546fb07a413fd30, State: Initialized, Role: FOLLOWER
I20260621 06:53:14.601162 31747 consensus_queue.cc:260] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d27db21ae2c4152b584ef4ac372d9e8" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36505 } } peers { permanent_uuid: "bbebee7b12a949d2b3b6ac148993faad" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36377 } } peers { permanent_uuid: "cdb716b66b8d4215a546fb07a413fd30" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41233 } }
I20260621 06:53:14.602171 31726 heartbeater.cc:503] Master 127.20.120.254:34091 was elected leader, sending a full tablet report...
I20260621 06:53:14.602785 31747 ts_tablet_manager.cc:1428] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
W20260621 06:53:14.726264 31727 tablet.cc:2369] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:53:14.749617 31594 tablet.cc:2369] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:53:14.757586 31460 tablet.cc:2369] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:53:14.802960 31752 raft_consensus.cc:482] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:53:14.803246 31752 raft_consensus.cc:504] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d27db21ae2c4152b584ef4ac372d9e8" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36505 } } peers { permanent_uuid: "bbebee7b12a949d2b3b6ac148993faad" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36377 } } peers { permanent_uuid: "cdb716b66b8d4215a546fb07a413fd30" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41233 } }
I20260621 06:53:14.804970 31752 leader_election.cc:288] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d27db21ae2c4152b584ef4ac372d9e8 (127.20.120.193:36505), bbebee7b12a949d2b3b6ac148993faad (127.20.120.194:36377)
I20260621 06:53:14.811919 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.805506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:59455 (local address 127.20.120.193:36505)
0621 06:53:14.805759 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:14.805768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:14.805797 (+ 29us) server_negotiation.cc:407] Connection header received
0621 06:53:14.806007 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:14.806014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:14.806095 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:14.806199 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:53:14.806702 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.807712 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.809975 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.811155 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.811326 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.811406 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:14.811570 (+ 164us) server_negotiation.cc:299] Negotiation successful
0621 06:53:14.811706 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260621 06:53:14.811928 31728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.805343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:36505 (local address 127.20.120.195:59455)
0621 06:53:14.805642 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:14.805661 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:14.805785 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:14.806257 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:14.806261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:14.806296 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:53:14.806517 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.806529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.807841 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.807844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.809770 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.809785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.811298 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.811302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.811425 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.811448 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 06:53:14.811520 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 06:53:14.811636 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260621 06:53:14.812546 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.805543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:46265 (local address 127.20.120.194:36377)
0621 06:53:14.805753 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:14.805758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:14.806438 (+ 680us) server_negotiation.cc:407] Connection header received
0621 06:53:14.806579 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:14.806583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:14.806692 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:14.806796 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:53:14.807224 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.808105 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.810270 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.811554 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.811697 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.811758 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:14.812150 (+ 392us) server_negotiation.cc:299] Negotiation successful
0621 06:53:14.812307 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 06:53:14.813110 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.805453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:36377 (local address 127.20.120.195:46265)
0621 06:53:14.806298 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:14.806314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:14.806438 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:14.806809 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:14.806812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:14.806830 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:53:14.807084 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.807090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.808242 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.808250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.810132 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:14.810141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.811766 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.811776 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:14.811952 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.811996 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 06:53:14.812751 (+ 755us) client_negotiation.cc:240] Negotiation successful
0621 06:53:14.812853 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":1,"thread_start_us":481,"threads_started":1}
I20260621 06:53:14.813086 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7683a11ee19841a0a1fbdde7be11897c" candidate_uuid: "cdb716b66b8d4215a546fb07a413fd30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d27db21ae2c4152b584ef4ac372d9e8" is_pre_election: true
I20260621 06:53:14.813683 31414 raft_consensus.cc:2454] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdb716b66b8d4215a546fb07a413fd30 in term 0.
I20260621 06:53:14.814136 31548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7683a11ee19841a0a1fbdde7be11897c" candidate_uuid: "cdb716b66b8d4215a546fb07a413fd30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbebee7b12a949d2b3b6ac148993faad" is_pre_election: true
I20260621 06:53:14.814378 31615 leader_election.cc:302] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d27db21ae2c4152b584ef4ac372d9e8, cdb716b66b8d4215a546fb07a413fd30; no voters:
I20260621 06:53:14.814687 31548 raft_consensus.cc:2454] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdb716b66b8d4215a546fb07a413fd30 in term 0.
I20260621 06:53:14.814805 31752 raft_consensus.cc:2781] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:53:14.814978 31752 raft_consensus.cc:482] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:53:14.815105 31752 raft_consensus.cc:3037] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:53:14.817366 31752 raft_consensus.cc:504] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d27db21ae2c4152b584ef4ac372d9e8" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36505 } } peers { permanent_uuid: "bbebee7b12a949d2b3b6ac148993faad" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36377 } } peers { permanent_uuid: "cdb716b66b8d4215a546fb07a413fd30" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41233 } }
I20260621 06:53:14.818053 31752 leader_election.cc:288] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [CANDIDATE]: Term 1 election: Requested vote from peers 2d27db21ae2c4152b584ef4ac372d9e8 (127.20.120.193:36505), bbebee7b12a949d2b3b6ac148993faad (127.20.120.194:36377)
I20260621 06:53:14.818444 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7683a11ee19841a0a1fbdde7be11897c" candidate_uuid: "cdb716b66b8d4215a546fb07a413fd30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d27db21ae2c4152b584ef4ac372d9e8"
I20260621 06:53:14.818565 31548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7683a11ee19841a0a1fbdde7be11897c" candidate_uuid: "cdb716b66b8d4215a546fb07a413fd30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbebee7b12a949d2b3b6ac148993faad"
I20260621 06:53:14.818681 31414 raft_consensus.cc:3037] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:53:14.818827 31548 raft_consensus.cc:3037] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:53:14.820844 31414 raft_consensus.cc:2454] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdb716b66b8d4215a546fb07a413fd30 in term 1.
I20260621 06:53:14.821211 31548 raft_consensus.cc:2454] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdb716b66b8d4215a546fb07a413fd30 in term 1.
I20260621 06:53:14.821225 31615 leader_election.cc:302] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d27db21ae2c4152b584ef4ac372d9e8, cdb716b66b8d4215a546fb07a413fd30; no voters:
I20260621 06:53:14.821518 31752 raft_consensus.cc:2781] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:53:14.822031 31752 raft_consensus.cc:686] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 1 LEADER]: Becoming Leader. State: Replica: cdb716b66b8d4215a546fb07a413fd30, State: Running, Role: LEADER
I20260621 06:53:14.822521 31752 consensus_queue.cc:237] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d27db21ae2c4152b584ef4ac372d9e8" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36505 } } peers { permanent_uuid: "bbebee7b12a949d2b3b6ac148993faad" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36377 } } peers { permanent_uuid: "cdb716b66b8d4215a546fb07a413fd30" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41233 } }
I20260621 06:53:14.827046 31270 catalog_manager.cc:5495] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdb716b66b8d4215a546fb07a413fd30 (127.20.120.195). New cstate: current_term: 1 leader_uuid: "cdb716b66b8d4215a546fb07a413fd30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d27db21ae2c4152b584ef4ac372d9e8" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbebee7b12a949d2b3b6ac148993faad" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdb716b66b8d4215a546fb07a413fd30" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41233 } health_report { overall_health: HEALTHY } } }
I20260621 06:53:14.850069 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.844475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.20.120.195:41233)
0621 06:53:14.844685 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:14.844691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:14.844711 (+ 20us) server_negotiation.cc:407] Connection header received
0621 06:53:14.844862 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:14.844866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:14.844949 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:14.845138 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:53:14.845530 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.846299 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.847770 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.848184 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.848337 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.848892 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:53:14.849491 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:53:14.849613 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:14.849684 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 06:53:14.849754 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 06:53:14.858791 31414 raft_consensus.cc:1261] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8 [term 1 FOLLOWER]: Refusing update from remote peer cdb716b66b8d4215a546fb07a413fd30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:53:14.858791 31548 raft_consensus.cc:1261] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad [term 1 FOLLOWER]: Refusing update from remote peer cdb716b66b8d4215a546fb07a413fd30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:53:14.859985 31752 consensus_queue.cc:1035] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d27db21ae2c4152b584ef4ac372d9e8" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:53:14.860340 31754 consensus_queue.cc:1035] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbebee7b12a949d2b3b6ac148993faad" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:53:14.871554 31759 mvcc.cc:205] Tried to move back new op lower bound from 7299173559729987584 to 7299173559596793856. Current Snapshot: MvccSnapshot[applied={T|T < 7299173559729987584}]
I20260621 06:53:14.888954 31269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:53:14.889253 31269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:53:14.892817 31269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af12b4e9f6a0478fb5154e7778f69e9e: Sending DeleteTablet for 3 replicas of tablet 7683a11ee19841a0a1fbdde7be11897c
I20260621 06:53:14.894027 31394 tablet_service.cc:1514] Processing DeleteTablet for tablet 7683a11ee19841a0a1fbdde7be11897c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:53:14 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:52146
I20260621 06:53:14.894027 31528 tablet_service.cc:1514] Processing DeleteTablet for tablet 7683a11ee19841a0a1fbdde7be11897c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:53:14 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:44492
I20260621 06:53:14.894924 31768 tablet_replica.cc:330] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8: stopping tablet replica
I20260621 06:53:14.895293 31768 raft_consensus.cc:2229] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:53:14.895481 31661 tablet_service.cc:1514] Processing DeleteTablet for tablet 7683a11ee19841a0a1fbdde7be11897c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:53:14 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:56564
I20260621 06:53:14.895733 31768 raft_consensus.cc:2258] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:53:14.896564 31767 tablet_replica.cc:330] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad: stopping tablet replica
I20260621 06:53:14.896925 31767 raft_consensus.cc:2229] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:53:14.897310 31767 raft_consensus.cc:2258] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:53:14.897495 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 31331
I20260621 06:53:14.899766 31768 ts_tablet_manager.cc:1905] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:53:14.906540 31769 tablet_replica.cc:330] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30: stopping tablet replica
I20260621 06:53:14.907184 31769 raft_consensus.cc:2229] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:53:14.907418 31767 ts_tablet_manager.cc:1905] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:53:14.908087 31769 raft_consensus.cc:2258] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:53:14.909780 31768 ts_tablet_manager.cc:1918] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:53:14.910063 31768 log.cc:1196] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/7683a11ee19841a0a1fbdde7be11897c
I20260621 06:53:14.910296 31769 ts_tablet_manager.cc:1905] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:53:14.910741 31768 ts_tablet_manager.cc:1939] T 7683a11ee19841a0a1fbdde7be11897c P 2d27db21ae2c4152b584ef4ac372d9e8: Deleting consensus metadata
I20260621 06:53:14.913007 31255 catalog_manager.cc:4841] TS 2d27db21ae2c4152b584ef4ac372d9e8 (127.20.120.193:36505): tablet 7683a11ee19841a0a1fbdde7be11897c (table test-table [id=3ca877a81d6844dfaea429790fb6b65b]) successfully deleted
I20260621 06:53:14.915925 31767 ts_tablet_manager.cc:1918] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:53:14.918260 31767 log.cc:1196] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/7683a11ee19841a0a1fbdde7be11897c
I20260621 06:53:14.918921 31767 ts_tablet_manager.cc:1939] T 7683a11ee19841a0a1fbdde7be11897c P bbebee7b12a949d2b3b6ac148993faad: Deleting consensus metadata
I20260621 06:53:14.920878 31257 catalog_manager.cc:4841] TS bbebee7b12a949d2b3b6ac148993faad (127.20.120.194:36377): tablet 7683a11ee19841a0a1fbdde7be11897c (table test-table [id=3ca877a81d6844dfaea429790fb6b65b]) successfully deleted
I20260621 06:53:14.923681 31769 ts_tablet_manager.cc:1918] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:53:14.923889 31769 log.cc:1196] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/7683a11ee19841a0a1fbdde7be11897c
I20260621 06:53:14.924376 31769 ts_tablet_manager.cc:1939] T 7683a11ee19841a0a1fbdde7be11897c P cdb716b66b8d4215a546fb07a413fd30: Deleting consensus metadata
I20260621 06:53:14.926100 31255 catalog_manager.cc:4841] TS cdb716b66b8d4215a546fb07a413fd30 (127.20.120.195:41233): tablet 7683a11ee19841a0a1fbdde7be11897c (table test-table [id=3ca877a81d6844dfaea429790fb6b65b]) successfully deleted
Jun 21 06:53:14 dist-test-slave-gmf2 krb5kdc[31225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:53:14.937637 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:14.898780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.20.120.193:36505)
0621 06:53:14.898956 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:14.898962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:14.915853 (+ 16891us) server_negotiation.cc:407] Connection header received
0621 06:53:14.916068 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:14.916075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:14.916165 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:14.916291 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:14.923092 (+ 6801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.924214 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.926448 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:14.927063 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:14.927190 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:14.931883 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:14.931898 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:14.931914 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:14.931955 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:14.934794 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:14.935498 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:14.935506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:14.935508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:14.935634 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:14.936032 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:14.936035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:14.936037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:14.936464 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:14.936776 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:14.937053 (+ 277us) server_negotiation.cc:299] Negotiation successful
0621 06:53:14.937254 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 06:53:14.938416 31454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:15.038923 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 31331
I20260621 06:53:15.058457 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 31465
Jun 21 06:53:15 dist-test-slave-gmf2 krb5kdc[31225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:53:15.076598 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:15.059325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.20.120.194:36377)
0621 06:53:15.059528 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:15.059534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:15.059557 (+ 23us) server_negotiation.cc:407] Connection header received
0621 06:53:15.059721 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:15.059728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:15.059824 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:15.059942 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:15.061229 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:15.062349 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:15.063673 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:15.064336 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:15.064468 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:15.069101 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:15.069121 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:15.069142 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:15.069185 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:15.072959 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:15.073606 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:15.073617 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:15.073621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:15.073773 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:15.074450 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:15.074458 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:15.074496 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:15.075036 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:15.075370 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:15.075980 (+ 610us) server_negotiation.cc:299] Negotiation successful
0621 06:53:15.076192 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 06:53:15.077448 31588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:15.195050 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 31465
I20260621 06:53:15.212641 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 31598
Jun 21 06:53:15 dist-test-slave-gmf2 krb5kdc[31225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:53:15.230810 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:15.213597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.20.120.195:41233)
0621 06:53:15.213808 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:15.213815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:15.213841 (+ 26us) server_negotiation.cc:407] Connection header received
0621 06:53:15.213937 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:15.213944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:15.214083 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:15.214204 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:15.215479 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:15.216751 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:15.218198 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:15.218925 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:15.219119 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:15.223324 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:15.223345 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:15.223370 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:15.223425 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:15.227717 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:15.228358 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:15.228369 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:15.228373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:15.228571 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:15.228944 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:15.228951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:15.228957 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:15.229502 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:15.229849 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:15.230078 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 06:53:15.230317 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 06:53:15.231616 31721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:15.353559 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 31598
I20260621 06:53:15.373134 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 31238
I20260621 06:53:15.374344 31300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:53:15.485735 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 31238
2026-06-21T06:53:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4308 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:53:15 dist-test-slave-gmf2 krb5kdc[31777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:53:15 dist-test-slave-gmf2 krb5kdc[31777](info): set up 2 sockets
Jun 21 06:53:15 dist-test-slave-gmf2 krb5kdc[31777](info): commencing operation
krb5kdc: starting...
W20260621 06:53:16.562865 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:53:16 dist-test-slave-gmf2 krb5kdc[31777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:53:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.120.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.120.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/rangeradmin_127.20.120.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/rangeradmin_127.20.120.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.120.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.120.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/rangerlookup_127.20.120.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/rangerlookup_127.20.120.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.120.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.120.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/HTTP_127.20.120.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/HTTP_127.20.120.212@KRBTEST.COM.keytab.
I20260621 06:53:16.875334 20963 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcf0MiJ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/postgres -l logfile start
2026-06-21 06:53:19.184 UTC [31807] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 06:53:19.184 UTC [31807] LOG: listening on IPv4 address "127.20.120.212", port 52415
2026-06-21 06:53:19.187 UTC [31807] LOG: listening on Unix socket "/tmp/.s.PGSQL.52415"
2026-06-21 06:53:19.209 UTC [31810] LOG: database system was shut down at 2026-06-21 06:53:17 UTC
2026-06-21 06:53:19.213 UTC [31807] LOG: database system is ready to accept connections
I20260621 06:53:20.182476 20963 mini_postgres.cc:96] Postgres bound to 52415
2026-06-21 06:53:20.199 UTC [31818] FATAL: database "slave" does not exist
127.20.120.212:52415 - accepting connections
I20260621 06:53:20.199893 20963 mini_ranger.cc:161] Starting Ranger...
I20260621 06:53:20.224702 20963 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 06:53:20.676270 20963 mini_ranger.cc:87] Created ranger Postgres database
I20260621 06:53:20.676376 20963 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-admin
2026-06-21 06:53:20,967 [I] DB FLAVOR :POSTGRES
2026-06-21 06:53:20,968 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 06:53:20,968 [I] Checking connection..
2026-06-21 06:53:20,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 06:53:21,197 [I] Checking connection passed.
2026-06-21 06:53:21,197 [I] --------- Verifying version history table ---------
2026-06-21 06:53:21,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 06:53:21,424 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 06:53:21,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 06:53:21,663 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 06:53:21,663 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 06:53:21,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 06:53:21.917 UTC [31882] WARNING: there is no transaction in progress
2026-06-21 06:53:21,923 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 06:53:21,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 06:53:22,180 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 06:53:22,180 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 06:53:22,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 06:53:22,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 06:53:22,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gmf2.c.gcp-upstream.internal','N') ;"
2026-06-21 06:53:22,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 06:53:23,142 [I] Table x_portal_user does not exist in database ranger
2026-06-21 06:53:23,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 06:53:23,400 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 06:53:23,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 06:53:23,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 06:53:23,848 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 06:53:23,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 06:53:24.749 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.765 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.780 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.795 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.813 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.830 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.882 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.888 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.901 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.910 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.920 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.933 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.942 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.949 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.955 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:24.960 UTC [31990] WARNING: there is no transaction in progress
2026-06-21 06:53:25,524 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 06:53:25,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 06:53:25,766 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 06:53:25,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 06:53:25,997 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 06:53:25,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 06:53:26,252 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 06:53:26,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 06:53:26,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 06:53:26,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gmf2.c.gcp-upstream.internal';"
2026-06-21 06:53:26,982 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 06:53:26,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 06:53:27,214 [I] DB_PATCHES have already been applied
I20260621 06:53:27.220633 20963 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcf0MiJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 06:53:27.220719 20963 mini_ranger.cc:193] Using host: 127.20.120.212
I20260621 06:53:27.235255 20963 mini_ranger.cc:227] Ranger admin URL: http://127.20.120.212:60883
Jun 21, 2026 6:53:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 6:53:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 6:53:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 6:53:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 6:53:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 6:53:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 6:53:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 06:53:28.351329 20963 mini_ranger.cc:160] Time spent starting Ranger: real 8.151s user 0.007s sys 0.028s
Jun 21, 2026 6:53:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 6:53:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 6:53:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 6:53:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 6:53:47 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 6:53:48 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 06:53:48.668634 20963 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.120.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.120.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/rangerkms_127.20.120.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/rangerkms_127.20.120.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.120.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.120.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.120.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/HTTP_127.20.120.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.120.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/HTTP_127.20.120.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 06:53:48 dist-test-slave-gmf2 krb5kdc[31777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024828, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 06:53:48.841377 20963 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 06:53:48.863420 20963 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 06:53:49.667171 20963 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 06:53:49.667339 20963 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-kms
I20260621 06:53:49.667466 20963 mini_ranger_kms.cc:227] Using postgres at 127.20.120.212:52415
2026-06-21 06:53:50,082 [I] DB FLAVOR :POSTGRES
2026-06-21 06:53:50,082 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 06:53:50,082 [I] Checking connection
2026-06-21 06:53:50,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 06:53:50,330 [I] connection success
2026-06-21 06:53:50,331 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 06:53:50,331 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 06:53:50,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 06:53:50,571 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 06:53:50,571 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 06:53:50,572 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 06:53:50,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:52415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 06:53:50,831 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 06:53:51.266769 20963 mini_ranger_kms.cc:310] Created kms service
I20260621 06:53:51.454530 20963 mini_ranger_kms.cc:326] Created kudu user
I20260621 06:53:51.500560 20963 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 06:53:51 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024828, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:53:51.858448 20963 mini_ranger_kms.cc:384] Added ranger policy
I20260621 06:53:51.858793 20963 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 06:53:51.858860 20963 mini_ranger_kms.cc:240] Using host: 127.20.120.212
I20260621 06:53:51.880276 20963 mini_ranger_kms.cc:276] Ranger KMS PID: 32170
I20260621 06:53:51.880486 20963 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.120.212:45767
Jun 21, 2026 6:53:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 6:53:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 6:53:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 6:53:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 6:53:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 6:53:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 21 06:53:54 dist-test-slave-gmf2 krb5kdc[31777](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782024834, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.120.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:53:54.152876 20963 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.312s user 0.111s sys 0.329s
I20260621 06:53:54.153122 20963 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 06:53:54.153312 20963 mini_ranger_kms.cc:401] 127.20.120.212:45767/kms/v1/keys
Jun 21 06:53:54 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024834, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.120.212@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
Jun 21 06:53:54 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024834, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.120.212@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
Jun 21 06:53:57 dist-test-slave-gmf2 krb5kdc[31777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:53:57.200747 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:37845
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:37845
--ranger_config_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.120.212:45767/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:53:57.535367 32221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:57.535643 32221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:57.535728 32221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:57.542712 32221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:53:57.542815 32221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:57.542860 32221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:57.542922 32221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:53:57.542963 32221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:53:57.552906 32221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.120.212:45767/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:37845
--ranger_config_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:37845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:57.554144 32221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:57.555301 32221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:57.567373 32226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:57.567339 32227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:57.567878 32221 server_base.cc:1034] running on GCE node
W20260621 06:53:57.567409 32229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:57.569419 32221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:57.570780 32221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:57.571966 32221 hybrid_clock.cc:648] HybridClock initialized: now 1782024837571919 us; error 39 us; skew 500 ppm
I20260621 06:53:57.572284 32221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:53:57 dist-test-slave-gmf2 krb5kdc[31777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024837, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:53:57.578405 32221 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:53:57.579710 32221 webserver.cc:460] Webserver started at http://127.20.120.254:44101/ using document root <none> and password file <none>
I20260621 06:53:57.580274 32221 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:57.580382 32221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:57.580684 32221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 06:53:57 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024837, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:53:57.628332 32221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "9f77ca9ab4dd4f0a88aec01dca91a6bc"
format_stamp: "Formatted at 2026-06-21 06:53:57 on dist-test-slave-gmf2"
server_key: "5dfde82e5b1641b443824ce52e45b30d"
server_key_iv: "1a21ded2521f6dd971474a8e4a130438"
server_key_version: "kuduclusterkey@0"
I20260621 06:53:57.629429 32221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "9f77ca9ab4dd4f0a88aec01dca91a6bc"
format_stamp: "Formatted at 2026-06-21 06:53:57 on dist-test-slave-gmf2"
server_key: "5dfde82e5b1641b443824ce52e45b30d"
server_key_iv: "1a21ded2521f6dd971474a8e4a130438"
server_key_version: "kuduclusterkey@0"
I20260621 06:53:57.636263 32221 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260621 06:53:57.655052 32236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:57.656900 32221 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260621 06:53:57.657081 32221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "9f77ca9ab4dd4f0a88aec01dca91a6bc"
format_stamp: "Formatted at 2026-06-21 06:53:57 on dist-test-slave-gmf2"
server_key: "5dfde82e5b1641b443824ce52e45b30d"
server_key_iv: "1a21ded2521f6dd971474a8e4a130438"
server_key_version: "kuduclusterkey@0"
I20260621 06:53:57.657323 32221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:57.704396 32221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:57.708472 32221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:57.708827 32221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:57.728274 32221 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:37845
I20260621 06:53:57.728355 32287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:37845 every 8 connection(s)
I20260621 06:53:57.729554 32221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:53:57.733778 32288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:53:57.739542 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 32221
I20260621 06:53:57.739746 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:53:57.740363 20963 external_mini_cluster.cc:1428] Setting key 77d7c204713c6b9e69a866cf046f9927
I20260621 06:53:57.747768 32288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc: Bootstrap starting.
I20260621 06:53:57.751503 32288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc: Neither blocks nor log segments found. Creating new log.
I20260621 06:53:57.752791 32288 log.cc:824] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc: Log is configured to *not* fsync() on all Append() calls
I20260621 06:53:57.760267 32288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc: No bootstrap required, opened a new log
Jun 21 06:53:57 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:53:57.771405 32288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f77ca9ab4dd4f0a88aec01dca91a6bc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37845 } }
I20260621 06:53:57.771906 32288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:53:57.772012 32288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f77ca9ab4dd4f0a88aec01dca91a6bc, State: Initialized, Role: FOLLOWER
I20260621 06:53:57.772756 32288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f77ca9ab4dd4f0a88aec01dca91a6bc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37845 } }
I20260621 06:53:57.773022 32288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:53:57.773197 32288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:53:57.773376 32288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:53:57.776573 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:57.742428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.20.120.254:37845)
0621 06:53:57.756973 (+ 14545us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:57.756989 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:57.757038 (+ 49us) server_negotiation.cc:407] Connection header received
0621 06:53:57.759291 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:57.759312 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:57.759700 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:57.760082 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:57.761407 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:57.763159 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:57.764550 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:57.765199 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:57.765378 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:57.769224 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:57.769231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:57.769249 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:57.769280 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:57.772439 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:57.773164 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:57.773173 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:57.773181 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:57.773377 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:57.773774 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:57.773780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:57.773784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:57.774397 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:57.774721 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:57.775187 (+ 466us) server_negotiation.cc:299] Negotiation successful
0621 06:53:57.775621 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14293,"thread_start_us":14476,"threads_started":1}
I20260621 06:53:57.776764 32288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f77ca9ab4dd4f0a88aec01dca91a6bc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37845 } }
I20260621 06:53:57.777354 32288 leader_election.cc:302] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f77ca9ab4dd4f0a88aec01dca91a6bc; no voters:
I20260621 06:53:57.778091 32288 leader_election.cc:288] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:53:57.778215 32293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:53:57.781046 32293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [term 1 LEADER]: Becoming Leader. State: Replica: 9f77ca9ab4dd4f0a88aec01dca91a6bc, State: Running, Role: LEADER
I20260621 06:53:57.781646 32293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f77ca9ab4dd4f0a88aec01dca91a6bc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37845 } }
I20260621 06:53:57.783663 32288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:53:57.784205 32295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f77ca9ab4dd4f0a88aec01dca91a6bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f77ca9ab4dd4f0a88aec01dca91a6bc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37845 } } }
I20260621 06:53:57.784595 32295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [sys.catalog]: This master's current role is: LEADER
I20260621 06:53:57.785043 32294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f77ca9ab4dd4f0a88aec01dca91a6bc. Latest consensus state: current_term: 1 leader_uuid: "9f77ca9ab4dd4f0a88aec01dca91a6bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f77ca9ab4dd4f0a88aec01dca91a6bc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37845 } } }
I20260621 06:53:57.785324 32294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc [sys.catalog]: This master's current role is: LEADER
I20260621 06:53:57.794430 32298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:53:57.799664 32298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:53:57.810505 32298 catalog_manager.cc:1347] Generated new cluster ID: 40f709bcf01b4f518e5353feb1f2156a
I20260621 06:53:57.810636 32298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:53:57.829306 32298 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:53:57.831875 32298 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:53:57.842232 32288 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 06:53:57.842428 32298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc: Generated new TSK 0
I20260621 06:53:57.843572 32298 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:53:58.673744 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.120.212:45767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:37845
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 06:53:58 dist-test-slave-gmf2 krb5kdc[31777](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782024838, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 06:53:59.082170 32334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:59.082614 32334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:59.082799 32334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:59.090314 32334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:59.090452 32334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:59.090612 32334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:53:59.100787 32334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.120.212:45767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:37845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:59.102325 32334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:59.103678 32334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:59.122663 32345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:59.123010 32347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:59.123571 32344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:59.124105 32334 server_base.cc:1034] running on GCE node
I20260621 06:53:59.124799 32334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:59.126000 32334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:59.127357 32334 hybrid_clock.cc:648] HybridClock initialized: now 1782024839127208 us; error 84 us; skew 500 ppm
I20260621 06:53:59.127764 32334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:53:59 dist-test-slave-gmf2 krb5kdc[31777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024839, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:53:59.132295 32334 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:53:59.147522 32334 webserver.cc:460] Webserver started at http://127.20.120.193:40429/ using document root <none> and password file <none>
I20260621 06:53:59.148924 32334 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:59.149092 32334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:59.149456 32334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 06:53:59 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024839, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:53:59.176182 32334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "f145fe169fb94c018f7895aeead5e5c5"
format_stamp: "Formatted at 2026-06-21 06:53:59 on dist-test-slave-gmf2"
server_key: "3b8743c4fde4454c4674d262604bec0b"
server_key_iv: "4e1ba1a277a491437fc38a406255d82f"
server_key_version: "kuduclusterkey@0"
I20260621 06:53:59.177397 32334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "f145fe169fb94c018f7895aeead5e5c5"
format_stamp: "Formatted at 2026-06-21 06:53:59 on dist-test-slave-gmf2"
server_key: "3b8743c4fde4454c4674d262604bec0b"
server_key_iv: "4e1ba1a277a491437fc38a406255d82f"
server_key_version: "kuduclusterkey@0"
I20260621 06:53:59.185307 32334 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260621 06:53:59.205317 32354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:59.207368 32334 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260621 06:53:59.207643 32334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "f145fe169fb94c018f7895aeead5e5c5"
format_stamp: "Formatted at 2026-06-21 06:53:59 on dist-test-slave-gmf2"
server_key: "3b8743c4fde4454c4674d262604bec0b"
server_key_iv: "4e1ba1a277a491437fc38a406255d82f"
server_key_version: "kuduclusterkey@0"
I20260621 06:53:59.208079 32334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:53:59.225756 32334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:53:59.230247 32334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:53:59.230547 32334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:53:59.231518 32334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:53:59.232930 32334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:53:59.233035 32334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:59.233178 32334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:53:59.233248 32334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:53:59.306886 32334 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:43541
I20260621 06:53:59.306952 32466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:43541 every 8 connection(s)
I20260621 06:53:59.308791 32334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:53:59.316216 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 32334
I20260621 06:53:59.316421 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:53:59.317224 20963 external_mini_cluster.cc:1428] Setting key 11ad69eed7ce6f666c5ef8484a61c621
Jun 21 06:53:59 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024839, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
I20260621 06:53:59.354887 32470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:59.312911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:47475 (local address 127.20.120.254:37845)
0621 06:53:59.329904 (+ 16993us) server_negotiation.cc:206] Beginning negotiation
0621 06:53:59.329911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:53:59.333285 (+ 3374us) server_negotiation.cc:407] Connection header received
0621 06:53:59.333354 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:53:59.333360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:53:59.333442 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:53:59.333565 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:53:59.335952 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:59.337135 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:59.339584 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:53:59.340190 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:53:59.340298 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:59.344837 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:53:59.344844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:53:59.344848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:53:59.344911 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:53:59.348586 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:59.349535 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:59.349541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:59.349544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:59.349696 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:53:59.350293 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:53:59.350300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:53:59.350306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:53:59.350673 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:53:59.350830 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 06:53:59.354119 (+ 3289us) server_negotiation.cc:299] Negotiation successful
0621 06:53:59.354396 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16856,"thread_start_us":16730,"threads_started":1}
I20260621 06:53:59.361001 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:53:59.312261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:37845 (local address 127.20.120.193:47475)
0621 06:53:59.330516 (+ 18255us) negotiation.cc:107] Waiting for socket to connect
0621 06:53:59.330571 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 06:53:59.332911 (+ 2340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:53:59.334079 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:53:59.334091 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:53:59.334831 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:53:59.335728 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:59.335755 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:59.337335 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:59.337341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:59.339404 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:53:59.339418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:53:59.340669 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:53:59.340675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:53:59.340936 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:53:59.341776 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:53:59.341804 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:53:59.344577 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:53:59.348745 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:53:59.348754 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:53:59.348772 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:53:59.349379 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:53:59.349858 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:53:59.349862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:53:59.349866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:53:59.350153 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:53:59.350916 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:53:59.350925 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:53:59.351392 (+ 467us) client_negotiation.cc:769] Sending connection context
0621 06:53:59.359612 (+ 8220us) client_negotiation.cc:240] Negotiation successful
0621 06:53:59.360017 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17927,"thread_start_us":17732,"threads_started":1}
I20260621 06:53:59.363296 32467 heartbeater.cc:347] Connected to a master server at 127.20.120.254:37845
I20260621 06:53:59.363627 32467 heartbeater.cc:464] Registering TS with master...
I20260621 06:53:59.364538 32467 heartbeater.cc:511] Master 127.20.120.254:37845 requested a full tablet report, sending...
I20260621 06:53:59.366772 32253 ts_manager.cc:194] Registered new tserver with Master: f145fe169fb94c018f7895aeead5e5c5 (127.20.120.193:43541)
I20260621 06:53:59.369019 32253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:47475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:53:59.389199 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.120.212:45767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:37845
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 06:53:59 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024838, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
Jun 21 06:53:59 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024838, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
W20260621 06:53:59.862311 32473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:53:59.862862 32473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:53:59.863080 32473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:53:59.873728 32473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:53:59.873965 32473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:53:59.874192 32473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:53:59.889072 32473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.120.212:45767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:37845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:53:59.890992 32473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:53:59.892681 32473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:53:59.911762 32480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:53:59.912591 32481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:59.933491 32473 server_base.cc:1034] running on GCE node
W20260621 06:53:59.934365 32483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:53:59.935258 32473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:53:59.941610 32473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:53:59.942904 32473 hybrid_clock.cc:648] HybridClock initialized: now 1782024839942779 us; error 123 us; skew 500 ppm
I20260621 06:53:59.943373 32473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:53:59 dist-test-slave-gmf2 krb5kdc[31777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024839, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:53:59.947784 32473 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:53:59.975214 32473 webserver.cc:460] Webserver started at http://127.20.120.194:38541/ using document root <none> and password file <none>
I20260621 06:53:59.976382 32473 fs_manager.cc:359] Metadata directory not provided
I20260621 06:53:59.976617 32473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:53:59.977031 32473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 06:53:59 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024839, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:54:00.004837 32473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "e5adcbcd69e7480699040047581ab6c7"
format_stamp: "Formatted at 2026-06-21 06:53:59 on dist-test-slave-gmf2"
server_key: "c74b9e6e5683f5e8183a26459d0f6fa1"
server_key_iv: "cd3b2c76857b7d9be320444ce6963f2e"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:00.006014 32473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "e5adcbcd69e7480699040047581ab6c7"
format_stamp: "Formatted at 2026-06-21 06:53:59 on dist-test-slave-gmf2"
server_key: "c74b9e6e5683f5e8183a26459d0f6fa1"
server_key_iv: "cd3b2c76857b7d9be320444ce6963f2e"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:00.019718 32473 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.007s
I20260621 06:54:00.085449 32490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:00.087752 32473 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.005s sys 0.006s
I20260621 06:54:00.088047 32473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "e5adcbcd69e7480699040047581ab6c7"
format_stamp: "Formatted at 2026-06-21 06:53:59 on dist-test-slave-gmf2"
server_key: "c74b9e6e5683f5e8183a26459d0f6fa1"
server_key_iv: "cd3b2c76857b7d9be320444ce6963f2e"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:00.088423 32473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:00.182488 32473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:00.189638 32473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:00.190030 32473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:00.192631 32473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:54:00.208534 32473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:54:00.208717 32473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:00.208902 32473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:54:00.209005 32473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:00.263053 32473 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:34867
I20260621 06:54:00.263077 32609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:34867 every 8 connection(s)
I20260621 06:54:00.264876 32473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:54:00.274016 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 32473
I20260621 06:54:00.274210 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:54:00.274677 20963 external_mini_cluster.cc:1428] Setting key ed61b4447ca9dfc232100c6fb725458b
Jun 21 06:54:00 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024839, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:00.286952 32612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:00.267743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:57727 (local address 127.20.120.254:37845)
0621 06:54:00.268202 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:00.268207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:00.269315 (+ 1108us) server_negotiation.cc:407] Connection header received
0621 06:54:00.271203 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:00.271209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:00.271308 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:00.271425 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:00.272837 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.273595 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.275938 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.276580 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.276718 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:00.280270 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:00.280276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:00.280280 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:00.280310 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:00.283143 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:00.283881 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:00.283903 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:00.283906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:00.284074 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:00.284597 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:00.284602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:00.284605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:00.284906 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:00.285062 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:00.286320 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0621 06:54:00.286563 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":256,"threads_started":1}
I20260621 06:54:00.288087 32613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:00.268385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:37845 (local address 127.20.120.194:57727)
0621 06:54:00.269161 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:00.269197 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:00.270938 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:00.271667 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:00.271675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:00.272083 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:00.272646 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:00.272681 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.273772 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.273778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:00.275769 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:00.275784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.276948 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.276953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:00.277153 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:00.277871 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:00.277896 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:00.280035 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:00.283288 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:00.283298 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:00.283316 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:00.283748 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:00.284261 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:00.284266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:00.284270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:00.284476 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:00.285034 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:00.285042 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:00.285784 (+ 742us) client_negotiation.cc:769] Sending connection context
0621 06:54:00.286185 (+ 401us) client_negotiation.cc:240] Negotiation successful
0621 06:54:00.286813 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":328,"threads_started":1}
I20260621 06:54:00.289526 32610 heartbeater.cc:347] Connected to a master server at 127.20.120.254:37845
I20260621 06:54:00.289847 32610 heartbeater.cc:464] Registering TS with master...
I20260621 06:54:00.290572 32610 heartbeater.cc:511] Master 127.20.120.254:37845 requested a full tablet report, sending...
I20260621 06:54:00.292016 32253 ts_manager.cc:194] Registered new tserver with Master: e5adcbcd69e7480699040047581ab6c7 (127.20.120.194:34867)
I20260621 06:54:00.292821 32253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:57727
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:54:00.335469 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.120.212:45767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:37845
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:54:00.375334 32467 heartbeater.cc:503] Master 127.20.120.254:37845 was elected leader, sending a full tablet report...
W20260621 06:54:00.665691 32616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:00.666018 32616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:00.666118 32616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:00.672914 32616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:00.673007 32616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:00.673115 32616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:54:00.681953 32616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.120.212:45767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:37845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:00.683115 32616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:00.684291 32616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:54:00.697839 32621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:00.697876 32622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:00.697927 32624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:00.698498 32616 server_base.cc:1034] running on GCE node
I20260621 06:54:00.699039 32616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:00.699888 32616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:00.703202 32616 hybrid_clock.cc:648] HybridClock initialized: now 1782024840703149 us; error 44 us; skew 500 ppm
I20260621 06:54:00.703481 32616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:54:00 dist-test-slave-gmf2 krb5kdc[31777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024840, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:00.707022 32616 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:54:00.708474 32616 webserver.cc:460] Webserver started at http://127.20.120.195:37953/ using document root <none> and password file <none>
I20260621 06:54:00.709067 32616 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:00.709196 32616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:54:00.709489 32616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 06:54:00 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024840, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:54:00.726646 32616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "2ef5689086a447f0b32bb65b5712d40f"
format_stamp: "Formatted at 2026-06-21 06:54:00 on dist-test-slave-gmf2"
server_key: "ce049b09e646dc09f4f4c4f9563212a0"
server_key_iv: "e3c2ba0c71be81d4f002f733836886ce"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:00.727736 32616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "2ef5689086a447f0b32bb65b5712d40f"
format_stamp: "Formatted at 2026-06-21 06:54:00 on dist-test-slave-gmf2"
server_key: "ce049b09e646dc09f4f4c4f9563212a0"
server_key_iv: "e3c2ba0c71be81d4f002f733836886ce"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:00.734570 32616 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260621 06:54:00.748771 32631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:00.750406 32616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:54:00.750587 32616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "2ef5689086a447f0b32bb65b5712d40f"
format_stamp: "Formatted at 2026-06-21 06:54:00 on dist-test-slave-gmf2"
server_key: "ce049b09e646dc09f4f4c4f9563212a0"
server_key_iv: "e3c2ba0c71be81d4f002f733836886ce"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:00.750816 32616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:00.779556 32616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:00.782974 32616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:00.783255 32616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:00.784343 32616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:54:00.785763 32616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:54:00.785861 32616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:00.785997 32616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:54:00.786075 32616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:00.818909 32616 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:35431
I20260621 06:54:00.818924 32743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:35431 every 8 connection(s)
I20260621 06:54:00.820191 32616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:54:00.829809 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 32616
I20260621 06:54:00.830003 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:54:00.830530 20963 external_mini_cluster.cc:1428] Setting key e42eb123cc6cf623dedeeed37c18388a
Jun 21 06:54:00 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024840, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:00.839198 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:00.823007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:35713 (local address 127.20.120.254:37845)
0621 06:54:00.823429 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:00.823433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:00.824380 (+ 947us) server_negotiation.cc:407] Connection header received
0621 06:54:00.826070 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:00.826073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:00.826142 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:00.826277 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:00.827570 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.828272 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.829726 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.830381 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.830504 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:00.833870 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:00.833875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:00.833891 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:00.833959 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:00.836696 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:00.837324 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:00.837327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:00.837329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:00.837439 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:00.837798 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:00.837801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:00.837803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:00.838096 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:00.838212 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:00.838774 (+ 562us) server_negotiation.cc:299] Negotiation successful
0621 06:54:00.838919 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":194,"threads_started":1}
I20260621 06:54:00.840083 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:00.823554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:37845 (local address 127.20.120.195:35713)
0621 06:54:00.824215 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:00.824251 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:00.825790 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:00.826484 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:00.826493 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:00.826865 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:00.827431 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:00.827448 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.828431 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.828434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:00.829597 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:00.829607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.830566 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.830571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:00.830858 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:00.831558 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:00.831580 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:00.833691 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:00.836806 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:00.836813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:00.836823 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:00.837230 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:00.837528 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:00.837531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:00.837533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:00.837709 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:00.838194 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:00.838200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:00.838476 (+ 276us) client_negotiation.cc:769] Sending connection context
0621 06:54:00.838687 (+ 211us) client_negotiation.cc:240] Negotiation successful
0621 06:54:00.839047 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":259,"threads_started":1}
I20260621 06:54:00.841382 32744 heartbeater.cc:347] Connected to a master server at 127.20.120.254:37845
I20260621 06:54:00.841667 32744 heartbeater.cc:464] Registering TS with master...
I20260621 06:54:00.842396 32744 heartbeater.cc:511] Master 127.20.120.254:37845 requested a full tablet report, sending...
I20260621 06:54:00.843829 32253 ts_manager.cc:194] Registered new tserver with Master: 2ef5689086a447f0b32bb65b5712d40f (127.20.120.195:35431)
I20260621 06:54:00.844674 32253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:35713
I20260621 06:54:00.845868 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:54:00.862931 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:00.850608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.20.120.254:37845)
0621 06:54:00.850756 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:00.850760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:00.851291 (+ 531us) server_negotiation.cc:407] Connection header received
0621 06:54:00.851409 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:00.851412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:00.851479 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:00.851560 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:00.852897 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.853566 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.855118 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.855631 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.855737 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:00.858070 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:00.858075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:00.858078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:00.858105 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:00.860752 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:00.861324 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:00.861328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:00.861330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:00.861431 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:00.861729 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:00.861731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:00.861734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:00.862051 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:00.862222 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:00.862504 (+ 282us) server_negotiation.cc:299] Negotiation successful
0621 06:54:00.862654 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 06:54:00.866559 32253 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:54:00.873425 32253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:54:00.897207 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:00.885675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43541 (local address 127.0.0.1:37628)
0621 06:54:00.886462 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:00.886502 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:00.886670 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:00.887950 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:00.887960 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:00.888003 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:54:00.888483 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:00.888498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.890701 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.890709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:00.893426 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:00.893445 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.896029 (+ 2584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.896037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:00.896292 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:00.896353 (+ 61us) client_negotiation.cc:769] Sending connection context
0621 06:54:00.896485 (+ 132us) client_negotiation.cc:240] Negotiation successful
0621 06:54:00.896697 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":354,"threads_started":1}
I20260621 06:54:00.898303 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:00.886005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.20.120.193:43541)
0621 06:54:00.886624 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:00.886634 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:00.886666 (+ 32us) server_negotiation.cc:407] Connection header received
0621 06:54:00.886951 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:00.886960 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:00.887438 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:00.887627 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:54:00.888696 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.890508 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.893635 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.895480 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.895675 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:00.895878 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:00.897623 (+ 1745us) server_negotiation.cc:299] Negotiation successful
0621 06:54:00.897837 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":295,"threads_started":1}
I20260621 06:54:00.903172 32402 tablet_service.cc:1467] Processing CreateTablet for tablet f152efda245b4186b547c1443e566282 (DEFAULT_TABLE table=test-table [id=857c166600004646aa2fe10313baa6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:54:00.904563 32402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f152efda245b4186b547c1443e566282. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:54:00.907326 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:00.887680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.20.120.195:35431)
0621 06:54:00.899653 (+ 11973us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:00.899664 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:00.899717 (+ 53us) server_negotiation.cc:407] Connection header received
0621 06:54:00.899853 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:00.899863 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:00.900155 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:00.901050 (+ 895us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:54:00.901743 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.903049 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.905048 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.906293 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.906471 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:00.906620 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:00.906858 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 06:54:00.906999 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11791,"thread_start_us":11644,"threads_started":1}
I20260621 06:54:00.907328 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:00.887307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:35431 (local address 127.0.0.1:53156)
0621 06:54:00.898673 (+ 11366us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:00.898692 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:00.898787 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:00.900697 (+ 1910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:00.900704 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:00.900746 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:54:00.900897 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:00.900907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.903207 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.903212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:00.904890 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:00.904903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.906479 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.906485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:00.906666 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:00.906701 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 06:54:00.906805 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 06:54:00.906938 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11262,"mutex_wait_us":1,"thread_start_us":13180,"threads_started":1}
I20260621 06:54:00.908187 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:00.886890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:34867 (local address 127.0.0.1:55478)
0621 06:54:00.896394 (+ 9504us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:00.896417 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:00.896541 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:00.901945 (+ 5404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:00.901949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:00.901973 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:54:00.902368 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:00.902378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.903966 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.903973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:00.905700 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:00.905711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.907508 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.907518 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:00.907728 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:00.907768 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 06:54:00.907857 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 06:54:00.907956 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9350,"thread_start_us":9737,"threads_started":1}
I20260621 06:54:00.909307 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:00.887109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.20.120.194:34867)
0621 06:54:00.901020 (+ 13911us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:00.901027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:00.901050 (+ 23us) server_negotiation.cc:407] Connection header received
0621 06:54:00.901145 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:00.901154 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:00.901740 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:00.901948 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:54:00.902559 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.903849 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.905848 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:00.907117 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:00.907312 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:00.908753 (+ 1441us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:00.908833 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 06:54:00.908968 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13755,"thread_start_us":13707,"threads_started":1}
I20260621 06:54:00.910738 32679 tablet_service.cc:1467] Processing CreateTablet for tablet f152efda245b4186b547c1443e566282 (DEFAULT_TABLE table=test-table [id=857c166600004646aa2fe10313baa6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:54:00.912135 32679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f152efda245b4186b547c1443e566282. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:54:00.912665 32545 tablet_service.cc:1467] Processing CreateTablet for tablet f152efda245b4186b547c1443e566282 (DEFAULT_TABLE table=test-table [id=857c166600004646aa2fe10313baa6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:54:00.914076 32545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f152efda245b4186b547c1443e566282. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:54:00.919047 32763 tablet_bootstrap.cc:492] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5: Bootstrap starting.
I20260621 06:54:00.922673 32763 tablet_bootstrap.cc:654] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5: Neither blocks nor log segments found. Creating new log.
I20260621 06:54:00.923678 32763 log.cc:824] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5: Log is configured to *not* fsync() on all Append() calls
I20260621 06:54:00.925758 32765 tablet_bootstrap.cc:492] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f: Bootstrap starting.
I20260621 06:54:00.926791 32763 tablet_bootstrap.cc:492] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5: No bootstrap required, opened a new log
I20260621 06:54:00.928087 32766 tablet_bootstrap.cc:492] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7: Bootstrap starting.
I20260621 06:54:00.928568 32763 ts_tablet_manager.cc:1397] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 06:54:00.930027 32765 tablet_bootstrap.cc:654] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f: Neither blocks nor log segments found. Creating new log.
I20260621 06:54:00.930743 32766 tablet_bootstrap.cc:654] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7: Neither blocks nor log segments found. Creating new log.
I20260621 06:54:00.931221 32765 log.cc:824] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f: Log is configured to *not* fsync() on all Append() calls
I20260621 06:54:00.931666 32766 log.cc:824] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7: Log is configured to *not* fsync() on all Append() calls
I20260621 06:54:00.934557 32765 tablet_bootstrap.cc:492] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f: No bootstrap required, opened a new log
I20260621 06:54:00.934813 32765 ts_tablet_manager.cc:1397] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 06:54:00.934991 32766 tablet_bootstrap.cc:492] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7: No bootstrap required, opened a new log
I20260621 06:54:00.935236 32766 ts_tablet_manager.cc:1397] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 06:54:00.937788 32763 raft_consensus.cc:348] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f145fe169fb94c018f7895aeead5e5c5" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43541 } } peers { permanent_uuid: "e5adcbcd69e7480699040047581ab6c7" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34867 } } peers { permanent_uuid: "2ef5689086a447f0b32bb65b5712d40f" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35431 } }
I20260621 06:54:00.938203 32763 raft_consensus.cc:374] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:54:00.938341 32763 raft_consensus.cc:729] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f145fe169fb94c018f7895aeead5e5c5, State: Initialized, Role: FOLLOWER
I20260621 06:54:00.938958 32763 consensus_queue.cc:260] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f145fe169fb94c018f7895aeead5e5c5" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43541 } } peers { permanent_uuid: "e5adcbcd69e7480699040047581ab6c7" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34867 } } peers { permanent_uuid: "2ef5689086a447f0b32bb65b5712d40f" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35431 } }
I20260621 06:54:00.943303 32763 ts_tablet_manager.cc:1428] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260621 06:54:00.945339 32765 raft_consensus.cc:348] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f145fe169fb94c018f7895aeead5e5c5" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43541 } } peers { permanent_uuid: "e5adcbcd69e7480699040047581ab6c7" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34867 } } peers { permanent_uuid: "2ef5689086a447f0b32bb65b5712d40f" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35431 } }
I20260621 06:54:00.945429 32766 raft_consensus.cc:348] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f145fe169fb94c018f7895aeead5e5c5" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43541 } } peers { permanent_uuid: "e5adcbcd69e7480699040047581ab6c7" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34867 } } peers { permanent_uuid: "2ef5689086a447f0b32bb65b5712d40f" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35431 } }
I20260621 06:54:00.945744 32765 raft_consensus.cc:374] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:54:00.945761 32766 raft_consensus.cc:374] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:54:00.945854 32766 raft_consensus.cc:729] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5adcbcd69e7480699040047581ab6c7, State: Initialized, Role: FOLLOWER
I20260621 06:54:00.945854 32765 raft_consensus.cc:729] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef5689086a447f0b32bb65b5712d40f, State: Initialized, Role: FOLLOWER
I20260621 06:54:00.946502 32765 consensus_queue.cc:260] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f145fe169fb94c018f7895aeead5e5c5" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43541 } } peers { permanent_uuid: "e5adcbcd69e7480699040047581ab6c7" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34867 } } peers { permanent_uuid: "2ef5689086a447f0b32bb65b5712d40f" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35431 } }
I20260621 06:54:00.946465 32766 consensus_queue.cc:260] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f145fe169fb94c018f7895aeead5e5c5" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43541 } } peers { permanent_uuid: "e5adcbcd69e7480699040047581ab6c7" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34867 } } peers { permanent_uuid: "2ef5689086a447f0b32bb65b5712d40f" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35431 } }
I20260621 06:54:00.948601 32610 heartbeater.cc:503] Master 127.20.120.254:37845 was elected leader, sending a full tablet report...
I20260621 06:54:00.949038 32744 heartbeater.cc:503] Master 127.20.120.254:37845 was elected leader, sending a full tablet report...
I20260621 06:54:00.949713 32765 ts_tablet_manager.cc:1428] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260621 06:54:00.950551 32766 ts_tablet_manager.cc:1428] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260621 06:54:01.000126 302 raft_consensus.cc:482] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:54:01.000427 302 raft_consensus.cc:504] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f145fe169fb94c018f7895aeead5e5c5" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43541 } } peers { permanent_uuid: "e5adcbcd69e7480699040047581ab6c7" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34867 } } peers { permanent_uuid: "2ef5689086a447f0b32bb65b5712d40f" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35431 } }
I20260621 06:54:01.002038 302 leader_election.cc:288] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f145fe169fb94c018f7895aeead5e5c5 (127.20.120.193:43541), 2ef5689086a447f0b32bb65b5712d40f (127.20.120.195:35431)
I20260621 06:54:01.008507 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:01.002638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:54321 (local address 127.20.120.195:35431)
0621 06:54:01.002876 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:01.002884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:01.003353 (+ 469us) server_negotiation.cc:407] Connection header received
0621 06:54:01.003591 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:01.003595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:01.003651 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:01.003746 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:54:01.004340 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.005036 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.006824 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.007582 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.007757 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:01.007830 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:01.008123 (+ 293us) server_negotiation.cc:299] Negotiation successful
0621 06:54:01.008245 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 06:54:01.008580 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:01.002445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:52641 (local address 127.20.120.193:43541)
0621 06:54:01.002711 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:01.002720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:01.003542 (+ 822us) server_negotiation.cc:407] Connection header received
0621 06:54:01.003729 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:01.003734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:01.003801 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:01.003895 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:54:01.004495 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.005178 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.006870 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.007672 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.007859 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:01.007930 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:01.008212 (+ 282us) server_negotiation.cc:299] Negotiation successful
0621 06:54:01.008335 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 06:54:01.008673 304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:01.002342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:35431 (local address 127.20.120.194:54321)
0621 06:54:01.003237 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:01.003252 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:01.003427 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:01.003764 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:01.003768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:01.003792 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:54:01.004191 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:01.004200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.005164 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.005170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:01.006674 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:01.006684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.007781 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.007787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:01.007961 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:01.007984 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 06:54:01.008098 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 06:54:01.008214 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":24,"spinlock_wait_cycles":17664,"thread_start_us":562,"threads_started":1}
I20260621 06:54:01.008700 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:01.002351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43541 (local address 127.20.120.194:52641)
0621 06:54:01.003425 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:01.003442 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:01.003582 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:01.003909 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:01.003913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:01.003949 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:54:01.004327 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:01.004335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.005314 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.005320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:01.006716 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:01.006726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.007885 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.007890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:01.008053 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:01.008074 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 06:54:01.008184 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 06:54:01.008297 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":4,"thread_start_us":757,"threads_started":1}
I20260621 06:54:01.010154 32699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f152efda245b4186b547c1443e566282" candidate_uuid: "e5adcbcd69e7480699040047581ab6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef5689086a447f0b32bb65b5712d40f" is_pre_election: true
I20260621 06:54:01.010156 32422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f152efda245b4186b547c1443e566282" candidate_uuid: "e5adcbcd69e7480699040047581ab6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f145fe169fb94c018f7895aeead5e5c5" is_pre_election: true
I20260621 06:54:01.010612 32422 raft_consensus.cc:2454] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5adcbcd69e7480699040047581ab6c7 in term 0.
I20260621 06:54:01.010599 32699 raft_consensus.cc:2454] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5adcbcd69e7480699040047581ab6c7 in term 0.
I20260621 06:54:01.011214 32493 leader_election.cc:302] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ef5689086a447f0b32bb65b5712d40f, e5adcbcd69e7480699040047581ab6c7, f145fe169fb94c018f7895aeead5e5c5; no voters:
I20260621 06:54:01.011579 302 raft_consensus.cc:2781] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:54:01.011721 302 raft_consensus.cc:482] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:54:01.011842 302 raft_consensus.cc:3037] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:54:01.014169 302 raft_consensus.cc:504] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f145fe169fb94c018f7895aeead5e5c5" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43541 } } peers { permanent_uuid: "e5adcbcd69e7480699040047581ab6c7" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34867 } } peers { permanent_uuid: "2ef5689086a447f0b32bb65b5712d40f" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35431 } }
I20260621 06:54:01.014755 302 leader_election.cc:288] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [CANDIDATE]: Term 1 election: Requested vote from peers f145fe169fb94c018f7895aeead5e5c5 (127.20.120.193:43541), 2ef5689086a447f0b32bb65b5712d40f (127.20.120.195:35431)
I20260621 06:54:01.015085 32422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f152efda245b4186b547c1443e566282" candidate_uuid: "e5adcbcd69e7480699040047581ab6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f145fe169fb94c018f7895aeead5e5c5"
I20260621 06:54:01.015316 32422 raft_consensus.cc:3037] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:54:01.015232 32699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f152efda245b4186b547c1443e566282" candidate_uuid: "e5adcbcd69e7480699040047581ab6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef5689086a447f0b32bb65b5712d40f"
I20260621 06:54:01.015462 32699 raft_consensus.cc:3037] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f [term 0 FOLLOWER]: Advancing to term 1
W20260621 06:54:01.016989 32611 tablet.cc:2369] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:54:01.017643 32699 raft_consensus.cc:2454] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5adcbcd69e7480699040047581ab6c7 in term 1.
I20260621 06:54:01.017887 32422 raft_consensus.cc:2454] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5adcbcd69e7480699040047581ab6c7 in term 1.
I20260621 06:54:01.018062 32491 leader_election.cc:302] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef5689086a447f0b32bb65b5712d40f, e5adcbcd69e7480699040047581ab6c7; no voters:
I20260621 06:54:01.018352 302 raft_consensus.cc:2781] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:54:01.018778 302 raft_consensus.cc:686] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 1 LEADER]: Becoming Leader. State: Replica: e5adcbcd69e7480699040047581ab6c7, State: Running, Role: LEADER
I20260621 06:54:01.019304 302 consensus_queue.cc:237] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f145fe169fb94c018f7895aeead5e5c5" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43541 } } peers { permanent_uuid: "e5adcbcd69e7480699040047581ab6c7" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34867 } } peers { permanent_uuid: "2ef5689086a447f0b32bb65b5712d40f" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35431 } }
I20260621 06:54:01.023937 32252 catalog_manager.cc:5495] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5adcbcd69e7480699040047581ab6c7 (127.20.120.194). New cstate: current_term: 1 leader_uuid: "e5adcbcd69e7480699040047581ab6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f145fe169fb94c018f7895aeead5e5c5" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5adcbcd69e7480699040047581ab6c7" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 34867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ef5689086a447f0b32bb65b5712d40f" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35431 } health_report { overall_health: UNKNOWN } } }
I20260621 06:54:01.061223 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:01.056138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.20.120.194:34867)
0621 06:54:01.056343 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:01.056348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:01.056366 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:54:01.056521 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:01.056525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:01.056595 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:01.056765 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:54:01.057141 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.057886 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.059317 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.059697 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.059824 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:01.060269 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:54:01.060710 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:54:01.060855 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:01.060905 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 06:54:01.060972 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 06:54:01.068637 32699 raft_consensus.cc:1261] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f [term 1 FOLLOWER]: Refusing update from remote peer e5adcbcd69e7480699040047581ab6c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:54:01.068637 32422 raft_consensus.cc:1261] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5 [term 1 FOLLOWER]: Refusing update from remote peer e5adcbcd69e7480699040047581ab6c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:54:01.069540 306 consensus_queue.cc:1035] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef5689086a447f0b32bb65b5712d40f" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:54:01.069818 302 consensus_queue.cc:1035] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f145fe169fb94c018f7895aeead5e5c5" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 06:54:01.072054 32745 tablet.cc:2369] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:54:01.074738 32468 tablet.cc:2369] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:54:01.096282 32253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:54:01.096539 32253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:54:01.100416 32253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f77ca9ab4dd4f0a88aec01dca91a6bc: Sending DeleteTablet for 3 replicas of tablet f152efda245b4186b547c1443e566282
I20260621 06:54:01.101855 32402 tablet_service.cc:1514] Processing DeleteTablet for tablet f152efda245b4186b547c1443e566282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:54:01 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:37628
I20260621 06:54:01.102321 32545 tablet_service.cc:1514] Processing DeleteTablet for tablet f152efda245b4186b547c1443e566282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:54:01 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:55478
I20260621 06:54:01.102432 32679 tablet_service.cc:1514] Processing DeleteTablet for tablet f152efda245b4186b547c1443e566282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:54:01 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:53156
I20260621 06:54:01.104564 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 32334
I20260621 06:54:01.114606 321 tablet_replica.cc:330] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f: stopping tablet replica
I20260621 06:54:01.115404 319 tablet_replica.cc:330] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5: stopping tablet replica
I20260621 06:54:01.116982 319 raft_consensus.cc:2229] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:54:01.117298 321 raft_consensus.cc:2229] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:54:01.118006 319 raft_consensus.cc:2258] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:54:01.118624 321 raft_consensus.cc:2258] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:54:01.119256 320 tablet_replica.cc:330] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7: stopping tablet replica
I20260621 06:54:01.120153 320 raft_consensus.cc:2229] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:54:01.121836 320 raft_consensus.cc:2258] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:54:01.122170 321 ts_tablet_manager.cc:1905] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 06:54:01 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:54:01.124832 320 ts_tablet_manager.cc:1905] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:54:01.126349 319 ts_tablet_manager.cc:1905] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:54:01.131700 321 ts_tablet_manager.cc:1918] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:54:01.132956 319 ts_tablet_manager.cc:1918] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:54:01.133095 320 ts_tablet_manager.cc:1918] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:54:01.133170 319 log.cc:1196] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/f152efda245b4186b547c1443e566282
I20260621 06:54:01.133386 320 log.cc:1196] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/f152efda245b4186b547c1443e566282
I20260621 06:54:01.133770 319 ts_tablet_manager.cc:1939] T f152efda245b4186b547c1443e566282 P f145fe169fb94c018f7895aeead5e5c5: Deleting consensus metadata
I20260621 06:54:01.133991 320 ts_tablet_manager.cc:1939] T f152efda245b4186b547c1443e566282 P e5adcbcd69e7480699040047581ab6c7: Deleting consensus metadata
I20260621 06:54:01.135656 32239 catalog_manager.cc:4841] TS f145fe169fb94c018f7895aeead5e5c5 (127.20.120.193:43541): tablet f152efda245b4186b547c1443e566282 (table test-table [id=857c166600004646aa2fe10313baa6b1]) successfully deleted
I20260621 06:54:01.135797 321 log.cc:1196] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/f152efda245b4186b547c1443e566282
I20260621 06:54:01.136006 32239 catalog_manager.cc:4841] TS e5adcbcd69e7480699040047581ab6c7 (127.20.120.194:34867): tablet f152efda245b4186b547c1443e566282 (table test-table [id=857c166600004646aa2fe10313baa6b1]) successfully deleted
I20260621 06:54:01.136308 321 ts_tablet_manager.cc:1939] T f152efda245b4186b547c1443e566282 P 2ef5689086a447f0b32bb65b5712d40f: Deleting consensus metadata
I20260621 06:54:01.138398 32237 catalog_manager.cc:4841] TS 2ef5689086a447f0b32bb65b5712d40f (127.20.120.195:35431): tablet f152efda245b4186b547c1443e566282 (table test-table [id=857c166600004646aa2fe10313baa6b1]) successfully deleted
I20260621 06:54:01.141008 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:01.106171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.20.120.193:43541)
0621 06:54:01.106363 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:01.106371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:01.109117 (+ 2746us) server_negotiation.cc:407] Connection header received
0621 06:54:01.109699 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:01.109709 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:01.109822 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:01.109926 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:01.112499 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.113721 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.115774 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.116479 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.116590 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:01.133240 (+ 16650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:01.133257 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:01.133277 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:01.133339 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:01.137676 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:01.138577 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:01.138586 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:01.138599 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:01.138759 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:01.139231 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:01.139235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:01.139239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:01.139720 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:01.139950 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:01.140391 (+ 441us) server_negotiation.cc:299] Negotiation successful
0621 06:54:01.140611 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 06:54:01.141714 32462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:01.238696 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 32334
I20260621 06:54:01.264110 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 32473
Jun 21 06:54:01 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:54:01.279816 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:01.264855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.20.120.194:34867)
0621 06:54:01.265044 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:01.265050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:01.265224 (+ 174us) server_negotiation.cc:407] Connection header received
0621 06:54:01.265417 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:01.265423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:01.265495 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:01.265599 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:01.266946 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.267835 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.269080 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.269535 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.269676 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:01.273958 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:01.273976 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:01.273992 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:01.274045 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:01.276958 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:01.277700 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:01.277709 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:01.277713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:01.277863 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:01.278322 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:01.278328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:01.278331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:01.278666 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:01.278975 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:01.279239 (+ 264us) server_negotiation.cc:299] Negotiation successful
0621 06:54:01.279429 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 06:54:01.280508 32605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:01.390357 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 32473
I20260621 06:54:01.407464 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 32616
Jun 21 06:54:01 dist-test-slave-gmf2 krb5kdc[31777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:54:01.421305 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:01.408283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.20.120.195:35431)
0621 06:54:01.408475 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:01.408479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:01.408496 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:54:01.408608 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:01.408612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:01.408670 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:01.408785 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:01.410098 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.410746 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.411897 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:01.412238 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:01.412320 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:01.416444 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:01.416456 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:01.416472 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:01.416506 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:01.419050 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:01.419576 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:01.419583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:01.419585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:01.419708 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:01.419994 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:01.419997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:01.419998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:01.420358 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:01.420553 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:01.420791 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 06:54:01.420932 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 06:54:01.421868 32739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:01.508749 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 32616
I20260621 06:54:01.524996 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 32221
I20260621 06:54:01.526188 32283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:01.636384 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 32221
I20260621 06:54:01.657868 20963 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 06:54:01.993607 20963 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 06:54:01.993875 20963 mini_ranger.cc:66] Stopping Ranger...
I20260621 06:54:02.338933 20963 mini_ranger.cc:68] Stopped Ranger
2026-06-21 06:54:02.339 UTC [31807] LOG: received smart shutdown request
2026-06-21 06:54:02.342 UTC [31807] LOG: background worker "logical replication launcher" (PID 31816) exited with exit code 1
2026-06-21 06:54:02.342 UTC [31811] LOG: shutting down
2026-06-21 06:54:02.495 UTC [31807] LOG: database system is shut down
2026-06-21T06:54:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47129 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:54:02 dist-test-slave-gmf2 krb5kdc[335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:54:02 dist-test-slave-gmf2 krb5kdc[335](info): set up 2 sockets
Jun 21 06:54:02 dist-test-slave-gmf2 krb5kdc[335](info): commencing operation
krb5kdc: starting...
W20260621 06:54:03.722046 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:54:03 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:54:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.120.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.120.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/rangeradmin_127.20.120.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/rangeradmin_127.20.120.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.120.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.120.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/rangerlookup_127.20.120.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/rangerlookup_127.20.120.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.120.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.120.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/HTTP_127.20.120.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/HTTP_127.20.120.212@KRBTEST.COM.keytab.
I20260621 06:54:04.010924 20963 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcf0MiJ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/postgres -l logfile start
2026-06-21 06:54:06.341 UTC [365] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 06:54:06.341 UTC [365] LOG: listening on IPv4 address "127.20.120.212", port 45435
2026-06-21 06:54:06.344 UTC [365] LOG: listening on Unix socket "/tmp/.s.PGSQL.45435"
2026-06-21 06:54:06.366 UTC [368] LOG: database system was shut down at 2026-06-21 06:54:05 UTC
2026-06-21 06:54:06.369 UTC [365] LOG: database system is ready to accept connections
I20260621 06:54:07.321993 20963 mini_postgres.cc:96] Postgres bound to 45435
2026-06-21 06:54:07.343 UTC [376] FATAL: database "slave" does not exist
127.20.120.212:45435 - accepting connections
I20260621 06:54:07.343633 20963 mini_ranger.cc:161] Starting Ranger...
I20260621 06:54:07.369576 20963 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 06:54:07.810942 20963 mini_ranger.cc:87] Created ranger Postgres database
I20260621 06:54:07.811077 20963 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-admin
2026-06-21 06:54:08,093 [I] DB FLAVOR :POSTGRES
2026-06-21 06:54:08,094 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 06:54:08,094 [I] Checking connection..
2026-06-21 06:54:08,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 06:54:08,340 [I] Checking connection passed.
2026-06-21 06:54:08,340 [I] --------- Verifying version history table ---------
2026-06-21 06:54:08,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 06:54:08,561 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 06:54:08,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 06:54:08,820 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 06:54:08,821 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 06:54:08,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 06:54:09.044 UTC [440] WARNING: there is no transaction in progress
2026-06-21 06:54:09,050 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 06:54:09,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 06:54:09,275 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 06:54:09,275 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 06:54:09,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 06:54:09,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 06:54:09,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gmf2.c.gcp-upstream.internal','N') ;"
2026-06-21 06:54:09,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 06:54:10,204 [I] Table x_portal_user does not exist in database ranger
2026-06-21 06:54:10,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 06:54:10,457 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 06:54:10,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 06:54:10,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 06:54:10,923 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 06:54:10,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 06:54:11.855 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:11.871 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:11.886 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:11.901 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:11.916 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:11.934 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:11.985 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:11.992 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:12.005 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:12.015 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:12.025 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:12.038 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:12.048 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:12.054 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:12.060 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:12.066 UTC [548] WARNING: there is no transaction in progress
2026-06-21 06:54:12,657 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 06:54:12,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 06:54:12,906 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 06:54:12,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 06:54:13,133 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 06:54:13,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 06:54:13,381 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 06:54:13,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 06:54:13,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 06:54:13,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gmf2.c.gcp-upstream.internal';"
2026-06-21 06:54:14,068 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 06:54:14,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 06:54:14,313 [I] DB_PATCHES have already been applied
I20260621 06:54:14.319638 20963 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcf0MiJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 06:54:14.319741 20963 mini_ranger.cc:193] Using host: 127.20.120.212
I20260621 06:54:14.334048 20963 mini_ranger.cc:227] Ranger admin URL: http://127.20.120.212:53003
Jun 21, 2026 6:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 6:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 6:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 6:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 6:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 6:54:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 6:54:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 06:54:15.436475 20963 mini_ranger.cc:160] Time spent starting Ranger: real 8.093s user 0.006s sys 0.032s
Jun 21, 2026 6:54:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 6:54:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 6:54:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 6:54:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 6:54:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 6:54:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 06:54:35.828560 20963 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.120.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.120.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/rangerkms_127.20.120.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.120.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/rangerkms_127.20.120.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.120.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.120.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.120.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/HTTP_127.20.120.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.120.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/HTTP_127.20.120.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 06:54:36 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024876, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 06:54:36.014792 20963 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 06:54:36.035267 20963 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 06:54:36.830858 20963 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 06:54:36.830983 20963 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-kms
I20260621 06:54:36.831035 20963 mini_ranger_kms.cc:227] Using postgres at 127.20.120.212:45435
2026-06-21 06:54:37,168 [I] DB FLAVOR :POSTGRES
2026-06-21 06:54:37,169 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 06:54:37,169 [I] Checking connection
2026-06-21 06:54:37,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 06:54:37,393 [I] connection success
2026-06-21 06:54:37,393 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 06:54:37,393 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 06:54:37,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 06:54:37,631 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 06:54:37,631 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 06:54:37,632 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 06:54:37,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.120.212:45435/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 06:54:37,873 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 06:54:38.334461 20963 mini_ranger_kms.cc:310] Created kms service
I20260621 06:54:38.528084 20963 mini_ranger_kms.cc:326] Created kudu user
I20260621 06:54:38.571250 20963 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 06:54:38 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024876, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:54:38.938565 20963 mini_ranger_kms.cc:384] Added ranger policy
I20260621 06:54:38.938897 20963 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcf0MiJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcf0MiJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 06:54:38.938958 20963 mini_ranger_kms.cc:240] Using host: 127.20.120.212
I20260621 06:54:38.956128 20963 mini_ranger_kms.cc:276] Ranger KMS PID: 728
I20260621 06:54:38.956262 20963 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.120.212:54817
Jun 21, 2026 6:54:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 6:54:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 6:54:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 6:54:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 6:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 6:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 21 06:54:41 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782024881, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.120.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:41.189379 20963 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.175s user 0.066s sys 0.290s
I20260621 06:54:41.189659 20963 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 06:54:41.189841 20963 mini_ranger_kms.cc:401] 127.20.120.212:54817/kms/v1/keys
Jun 21 06:54:41 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024881, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.120.212@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
Jun 21 06:54:41 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024881, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.120.212@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
Jun 21 06:54:44 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:54:44.169270 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:32873
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:32873
--ranger_config_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.120.212:54817/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:54:44.488900 779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:44.489178 779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:44.489321 779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:44.496862 779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:54:44.497009 779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:44.497092 779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:44.497189 779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:54:44.497272 779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:54:44.507257 779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.120.212:54817/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:32873
--ranger_config_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:44.508545 779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:44.509842 779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:54:44.523345 787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:44.523449 784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:44.523452 785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:44.524120 779 server_base.cc:1034] running on GCE node
I20260621 06:54:44.524914 779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:44.526387 779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:44.527578 779 hybrid_clock.cc:648] HybridClock initialized: now 1782024884527526 us; error 50 us; skew 500 ppm
I20260621 06:54:44.527882 779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:54:44 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024884, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:44.530894 779 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:54:44.532249 779 webserver.cc:460] Webserver started at http://127.20.120.254:43379/ using document root <none> and password file <none>
I20260621 06:54:44.532843 779 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:44.532946 779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:54:44.533226 779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 06:54:44 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024884, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:54:44.591485 779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "495dba44a6c643c19c67782f7a978808"
format_stamp: "Formatted at 2026-06-21 06:54:44 on dist-test-slave-gmf2"
server_key: "e3134fb6eeb68c94222993b2c86f21d9"
server_key_iv: "82315ee5f64fcb009fec79ccd6b8f8cd"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:44.592406 779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "495dba44a6c643c19c67782f7a978808"
format_stamp: "Formatted at 2026-06-21 06:54:44 on dist-test-slave-gmf2"
server_key: "e3134fb6eeb68c94222993b2c86f21d9"
server_key_iv: "82315ee5f64fcb009fec79ccd6b8f8cd"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:44.598090 779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260621 06:54:44.619311 794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:44.621013 779 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260621 06:54:44.621264 779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "495dba44a6c643c19c67782f7a978808"
format_stamp: "Formatted at 2026-06-21 06:54:44 on dist-test-slave-gmf2"
server_key: "e3134fb6eeb68c94222993b2c86f21d9"
server_key_iv: "82315ee5f64fcb009fec79ccd6b8f8cd"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:44.621528 779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:44.635087 779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:44.639120 779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:44.639430 779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:44.657378 779 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:32873
I20260621 06:54:44.657397 845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:32873 every 8 connection(s)
I20260621 06:54:44.658671 779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:54:44.662204 846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:54:44.664805 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 779
I20260621 06:54:44.665040 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:54:44.665835 20963 external_mini_cluster.cc:1428] Setting key c939659cc49ca6be0803b998e2450bf3
I20260621 06:54:44.674191 846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808: Bootstrap starting.
I20260621 06:54:44.677820 846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808: Neither blocks nor log segments found. Creating new log.
I20260621 06:54:44.679095 846 log.cc:824] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808: Log is configured to *not* fsync() on all Append() calls
Jun 21 06:54:44 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:44.682111 846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808: No bootstrap required, opened a new log
I20260621 06:54:44.688215 849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:44.667845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.20.120.254:32873)
0621 06:54:44.668616 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:44.668628 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:44.668670 (+ 42us) server_negotiation.cc:407] Connection header received
0621 06:54:44.670276 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:44.670297 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:44.670642 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:44.670944 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:44.672533 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:44.673760 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:44.675577 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:44.676232 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:44.676362 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:44.681200 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:44.681207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:44.681234 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:44.681270 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:44.684341 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:44.685102 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:44.685108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:44.685113 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:44.685261 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:44.685658 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:44.685661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:44.685662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:44.686188 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:44.686428 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:44.686892 (+ 464us) server_negotiation.cc:299] Negotiation successful
0621 06:54:44.687233 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":377,"threads_started":1}
I20260621 06:54:44.690937 846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495dba44a6c643c19c67782f7a978808" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32873 } }
I20260621 06:54:44.691356 846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:54:44.691464 846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495dba44a6c643c19c67782f7a978808, State: Initialized, Role: FOLLOWER
I20260621 06:54:44.691999 846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495dba44a6c643c19c67782f7a978808" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32873 } }
I20260621 06:54:44.692229 846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:54:44.692370 846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:54:44.692539 846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:54:44.694769 846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495dba44a6c643c19c67782f7a978808" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32873 } }
I20260621 06:54:44.695191 846 leader_election.cc:302] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 495dba44a6c643c19c67782f7a978808; no voters:
I20260621 06:54:44.695734 846 leader_election.cc:288] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:54:44.695819 851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:54:44.697433 846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:54:44.697763 851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 1 LEADER]: Becoming Leader. State: Replica: 495dba44a6c643c19c67782f7a978808, State: Running, Role: LEADER
I20260621 06:54:44.698289 851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495dba44a6c643c19c67782f7a978808" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32873 } }
I20260621 06:54:44.701762 852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "495dba44a6c643c19c67782f7a978808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495dba44a6c643c19c67782f7a978808" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32873 } } }
I20260621 06:54:44.701601 853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 495dba44a6c643c19c67782f7a978808. Latest consensus state: current_term: 1 leader_uuid: "495dba44a6c643c19c67782f7a978808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495dba44a6c643c19c67782f7a978808" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32873 } } }
I20260621 06:54:44.702157 852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [sys.catalog]: This master's current role is: LEADER
I20260621 06:54:44.702164 853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [sys.catalog]: This master's current role is: LEADER
I20260621 06:54:44.706881 862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:54:44.711943 862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:54:44.712822 846 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 06:54:44.737282 862 catalog_manager.cc:1347] Generated new cluster ID: e3b4892d4ca74b4bab4391a80d580dce
I20260621 06:54:44.737483 862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:54:44.780825 862 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:54:44.782837 862 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:54:44.793164 862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808: Generated new TSK 0
I20260621 06:54:44.794257 862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:54:45.524495 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.120.212:54817/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:32873
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 06:54:45 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782024885, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 06:54:45.891593 891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:45.892052 891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:45.892263 891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:45.900516 891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:45.900669 891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:45.900884 891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:54:45.911787 891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.120.212:54817/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:45.913237 891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:45.914870 891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:54:45.933646 901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:45.934461 902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:45.943347 891 server_base.cc:1034] running on GCE node
W20260621 06:54:45.943708 904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:45.944454 891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:45.945685 891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:45.946959 891 hybrid_clock.cc:648] HybridClock initialized: now 1782024885946885 us; error 71 us; skew 500 ppm
I20260621 06:54:45.947341 891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:54:45 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024885, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:45.950688 891 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:54:45.962379 891 webserver.cc:460] Webserver started at http://127.20.120.193:44119/ using document root <none> and password file <none>
I20260621 06:54:45.963263 891 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:45.963428 891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:54:45.963809 891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 06:54:45 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024885, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:54:45.990785 891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "209bcec7c7df4e199021233a7c13798e"
format_stamp: "Formatted at 2026-06-21 06:54:45 on dist-test-slave-gmf2"
server_key: "5deeedd9287b8d9b9223a2b66e4d0013"
server_key_iv: "5695bb4075bf31269615ce9c82695c34"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:45.993029 891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "209bcec7c7df4e199021233a7c13798e"
format_stamp: "Formatted at 2026-06-21 06:54:45 on dist-test-slave-gmf2"
server_key: "5deeedd9287b8d9b9223a2b66e4d0013"
server_key_iv: "5695bb4075bf31269615ce9c82695c34"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:46.003504 891 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.008s sys 0.000s
I20260621 06:54:46.038554 911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:46.040647 891 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.012s sys 0.004s
I20260621 06:54:46.040894 891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "209bcec7c7df4e199021233a7c13798e"
format_stamp: "Formatted at 2026-06-21 06:54:45 on dist-test-slave-gmf2"
server_key: "5deeedd9287b8d9b9223a2b66e4d0013"
server_key_iv: "5695bb4075bf31269615ce9c82695c34"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:46.041268 891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:46.074734 891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:46.079753 891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:46.080087 891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:46.081224 891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:54:46.082955 891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:54:46.083052 891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:46.083177 891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:54:46.083237 891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:46.141611 891 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:46031
I20260621 06:54:46.141629 1023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:46031 every 8 connection(s)
I20260621 06:54:46.143069 891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:54:46.150687 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 891
I20260621 06:54:46.150882 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:54:46.151649 20963 external_mini_cluster.cc:1428] Setting key 77c4c7f30251a7b1b809889c44672a39
Jun 21 06:54:46 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024885, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:46.180541 1026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:46.148573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:32873 (local address 127.20.120.193:44959)
0621 06:54:46.157545 (+ 8972us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:46.157604 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:46.160016 (+ 2412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:46.161125 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:46.161138 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:46.161821 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:46.162707 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:46.162735 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:46.164253 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:46.164258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:46.166378 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:46.166392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:46.167593 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:46.167601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:46.167911 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:46.168712 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:46.168743 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:46.171615 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:46.175746 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:46.175755 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:46.175772 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:46.176337 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:46.176783 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:46.176788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:46.176792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:46.177058 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:46.177790 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:46.177799 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:46.178269 (+ 470us) client_negotiation.cc:769] Sending connection context
0621 06:54:46.178571 (+ 302us) client_negotiation.cc:240] Negotiation successful
0621 06:54:46.179111 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8660,"thread_start_us":8455,"threads_started":1}
I20260621 06:54:46.181771 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:46.149823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:44959 (local address 127.20.120.254:32873)
0621 06:54:46.156947 (+ 7124us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:46.156957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:46.159220 (+ 2263us) server_negotiation.cc:407] Connection header received
0621 06:54:46.160453 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:46.160459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:46.160557 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:46.160665 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:46.162947 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:46.164075 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:46.166536 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:46.167141 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:46.167269 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:46.171897 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:46.171905 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:46.171910 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:46.171952 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:46.175558 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:46.176485 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:46.176490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:46.176493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:46.176648 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:46.177204 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:46.177209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:46.177213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:46.177572 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:46.177718 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:46.181159 (+ 3441us) server_negotiation.cc:299] Negotiation successful
0621 06:54:46.181379 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6969,"thread_start_us":6853,"threads_started":1}
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
I20260621 06:54:46.184067 1024 heartbeater.cc:347] Connected to a master server at 127.20.120.254:32873
I20260621 06:54:46.184500 1024 heartbeater.cc:464] Registering TS with master...
I20260621 06:54:46.185467 1024 heartbeater.cc:511] Master 127.20.120.254:32873 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
I20260621 06:54:46.188257 811 ts_manager.cc:194] Registered new tserver with Master: 209bcec7c7df4e199021233a7c13798e (127.20.120.193:46031)
I20260621 06:54:46.190321 811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:44959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:54:46.220798 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.120.212:54817/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:32873
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 06:54:46 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024885, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
Jun 21 06:54:46 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024885, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
W20260621 06:54:46.642373 1030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:46.642880 1030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:46.643095 1030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:46.654124 1030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:46.654263 1030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:46.654443 1030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:54:46.669222 1030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.120.212:54817/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:46.671000 1030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:46.672485 1030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:54:46.692072 1037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:46.692066 1038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:46.693503 1041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:46.694228 1030 server_base.cc:1034] running on GCE node
I20260621 06:54:46.695013 1030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:46.706120 1030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:46.707427 1030 hybrid_clock.cc:648] HybridClock initialized: now 1782024886707339 us; error 71 us; skew 500 ppm
I20260621 06:54:46.707813 1030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:54:46 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024886, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:46.715153 1030 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:54:46.734331 1030 webserver.cc:460] Webserver started at http://127.20.120.194:34111/ using document root <none> and password file <none>
I20260621 06:54:46.735306 1030 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:46.735483 1030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:54:46.735838 1030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 06:54:46 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024886, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:54:46.757690 1030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "e1b05c36cdb847fe8e59b312bb579812"
format_stamp: "Formatted at 2026-06-21 06:54:46 on dist-test-slave-gmf2"
server_key: "cfa4ac12d3d4daa956ed9248cc306fa8"
server_key_iv: "b6d530d46bccff265614e3fdc5c46e80"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:46.758766 1030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "e1b05c36cdb847fe8e59b312bb579812"
format_stamp: "Formatted at 2026-06-21 06:54:46 on dist-test-slave-gmf2"
server_key: "cfa4ac12d3d4daa956ed9248cc306fa8"
server_key_iv: "b6d530d46bccff265614e3fdc5c46e80"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:46.766379 1030 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260621 06:54:46.796864 1047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:46.799286 1030 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.002s
I20260621 06:54:46.799571 1030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "e1b05c36cdb847fe8e59b312bb579812"
format_stamp: "Formatted at 2026-06-21 06:54:46 on dist-test-slave-gmf2"
server_key: "cfa4ac12d3d4daa956ed9248cc306fa8"
server_key_iv: "b6d530d46bccff265614e3fdc5c46e80"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:46.800069 1030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:46.832336 1030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:46.837520 1030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:46.837845 1030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:46.838996 1030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:54:46.840878 1030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:54:46.840971 1030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:46.841086 1030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:54:46.841150 1030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:46.904606 1030 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:38341
I20260621 06:54:46.904629 1166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:38341 every 8 connection(s)
I20260621 06:54:46.906039 1030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:54:46.913311 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 1030
I20260621 06:54:46.913514 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:54:46.913995 20963 external_mini_cluster.cc:1428] Setting key e58e8638f9fef0837cc7b862e61a4582
Jun 21 06:54:46 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024886, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:46.934566 1169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:46.912485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:40611 (local address 127.20.120.254:32873)
0621 06:54:46.912942 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:46.912947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:46.915634 (+ 2687us) server_negotiation.cc:407] Connection header received
0621 06:54:46.918319 (+ 2685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:46.918325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:46.918417 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:46.918548 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:46.920575 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:46.921669 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:46.924538 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:46.925051 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:46.925194 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:46.928784 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:46.928790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:46.928797 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:46.928841 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:46.931500 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:46.932194 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:46.932200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:46.932204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:46.932330 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:46.932779 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:46.932784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:46.932790 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:46.933080 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:46.933238 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:46.933950 (+ 712us) server_negotiation.cc:299] Negotiation successful
0621 06:54:46.934184 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":240,"threads_started":1}
I20260621 06:54:46.935879 1170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:46.913262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:32873 (local address 127.20.120.194:40611)
0621 06:54:46.915469 (+ 2207us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:46.915521 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:46.917993 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:46.918826 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:46.918840 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:46.919497 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:46.920366 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:46.920394 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:46.921862 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:46.921870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:46.924035 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:46.924052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:46.925299 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:46.925315 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:46.925550 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:46.926356 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:46.926410 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:46.928556 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:46.931643 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:46.931653 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:46.931670 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:46.932067 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:46.932437 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:46.932462 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:46.932465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:46.932659 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:46.933210 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:46.933218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:46.933552 (+ 334us) client_negotiation.cc:769] Sending connection context
0621 06:54:46.933823 (+ 271us) client_negotiation.cc:240] Negotiation successful
0621 06:54:46.934416 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1867,"thread_start_us":1647,"threads_started":1}
I20260621 06:54:46.937439 1167 heartbeater.cc:347] Connected to a master server at 127.20.120.254:32873
I20260621 06:54:46.937762 1167 heartbeater.cc:464] Registering TS with master...
I20260621 06:54:46.938524 1167 heartbeater.cc:511] Master 127.20.120.254:32873 requested a full tablet report, sending...
I20260621 06:54:46.939930 811 ts_manager.cc:194] Registered new tserver with Master: e1b05c36cdb847fe8e59b312bb579812 (127.20.120.194:38341)
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
I20260621 06:54:46.940773 811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:40611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:54:46.973321 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.120.212:54817/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:32873
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:54:47.194003 1024 heartbeater.cc:503] Master 127.20.120.254:32873 was elected leader, sending a full tablet report...
W20260621 06:54:47.339782 1173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:47.340095 1173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:47.340260 1173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:47.347505 1173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:47.347681 1173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:47.347883 1173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:54:47.357617 1173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.120.212:54817/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:47.359088 1173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:47.360404 1173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:54:47.373003 1179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:47.373003 1181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:47.373067 1178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:47.379185 1173 server_base.cc:1034] running on GCE node
I20260621 06:54:47.379762 1173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:47.380748 1173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:47.381989 1173 hybrid_clock.cc:648] HybridClock initialized: now 1782024887381928 us; error 50 us; skew 500 ppm
I20260621 06:54:47.382308 1173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:54:47 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024887, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:47.385378 1173 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:54:47.386827 1173 webserver.cc:460] Webserver started at http://127.20.120.195:33041/ using document root <none> and password file <none>
I20260621 06:54:47.387636 1173 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:47.387780 1173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:54:47.388087 1173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 06:54:47 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024887, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:54:47.405602 1173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "97f432eeeb1f4e5a9e4479284b48fde3"
format_stamp: "Formatted at 2026-06-21 06:54:47 on dist-test-slave-gmf2"
server_key: "6647d7190922687a215ca726a2d8c426"
server_key_iv: "4de647b4198db8f011dccc6f1e308b9a"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:47.406783 1173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "97f432eeeb1f4e5a9e4479284b48fde3"
format_stamp: "Formatted at 2026-06-21 06:54:47 on dist-test-slave-gmf2"
server_key: "6647d7190922687a215ca726a2d8c426"
server_key_iv: "4de647b4198db8f011dccc6f1e308b9a"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:47.413779 1173 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260621 06:54:47.431326 1188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:47.432803 1173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260621 06:54:47.433007 1173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "97f432eeeb1f4e5a9e4479284b48fde3"
format_stamp: "Formatted at 2026-06-21 06:54:47 on dist-test-slave-gmf2"
server_key: "6647d7190922687a215ca726a2d8c426"
server_key_iv: "4de647b4198db8f011dccc6f1e308b9a"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:47.433246 1173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:47.478201 1173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:47.481760 1173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:47.482097 1173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:47.483003 1173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:54:47.484293 1173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:54:47.484357 1173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:47.484508 1173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:54:47.484555 1173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:47.516232 1173 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:40397
I20260621 06:54:47.516268 1300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:40397 every 8 connection(s)
I20260621 06:54:47.517580 1173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:54:47.518088 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 1173
I20260621 06:54:47.518283 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:54:47.518774 20963 external_mini_cluster.cc:1428] Setting key 4c6dfd33230842500b768d0c88f2ee0c
Jun 21 06:54:47 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024887, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:47.539278 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:47.520498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:41127 (local address 127.20.120.254:32873)
0621 06:54:47.521034 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:47.521038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:47.522213 (+ 1175us) server_negotiation.cc:407] Connection header received
0621 06:54:47.523962 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:47.523965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:47.524031 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:47.524119 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:47.525554 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:47.526375 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:47.528690 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:47.529339 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:47.529504 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:47.533464 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:47.533469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:47.533472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:47.533504 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:47.536283 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:47.536972 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:47.536975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:47.536978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:47.537114 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:47.537518 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:47.537524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:47.537530 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:47.537820 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:47.537997 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:47.538687 (+ 690us) server_negotiation.cc:299] Negotiation successful
0621 06:54:47.538901 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":235,"threads_started":1}
I20260621 06:54:47.540264 1304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:47.521197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:32873 (local address 127.20.120.195:41127)
0621 06:54:47.522040 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:47.522098 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:47.523723 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:47.524306 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:47.524326 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:47.524715 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:47.525352 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:47.525397 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:47.526548 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:47.526554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:47.528485 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:47.528500 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:47.529632 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:47.529641 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:47.529967 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:47.530831 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:47.530853 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:47.533228 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:47.536430 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:47.536443 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:47.536456 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:47.536881 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:47.537205 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:47.537208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:47.537211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:47.537387 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:47.537997 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:47.538006 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:47.538332 (+ 326us) client_negotiation.cc:769] Sending connection context
0621 06:54:47.538628 (+ 296us) client_negotiation.cc:240] Negotiation successful
0621 06:54:47.539176 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":300,"threads_started":1}
I20260621 06:54:47.541514 1301 heartbeater.cc:347] Connected to a master server at 127.20.120.254:32873
I20260621 06:54:47.541767 1301 heartbeater.cc:464] Registering TS with master...
I20260621 06:54:47.542459 1301 heartbeater.cc:511] Master 127.20.120.254:32873 requested a full tablet report, sending...
I20260621 06:54:47.543735 811 ts_manager.cc:194] Registered new tserver with Master: 97f432eeeb1f4e5a9e4479284b48fde3 (127.20.120.195:40397)
I20260621 06:54:47.544509 811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:41127
I20260621 06:54:47.546834 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:54:47.548609 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 891
Jun 21 06:54:47 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:54:47.563899 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:47.549380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.20.120.193:46031)
0621 06:54:47.549988 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:47.549996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:47.550028 (+ 32us) server_negotiation.cc:407] Connection header received
0621 06:54:47.550081 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:47.550087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:47.550254 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:47.550383 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:47.551706 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:47.552669 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:47.554200 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:47.554709 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:47.554854 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:47.558885 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:47.558896 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:47.558909 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:47.558945 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:47.561383 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:47.562197 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:47.562204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:47.562206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:47.562340 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:47.562669 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:47.562672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:47.562674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:47.562973 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:47.563185 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:47.563428 (+ 243us) server_negotiation.cc:299] Negotiation successful
0621 06:54:47.563636 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":270,"threads_started":1}
I20260621 06:54:47.565124 1019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:47.647847 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 891
I20260621 06:54:47.664654 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 1030
Jun 21 06:54:47 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:54:47.680734 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:47.665544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.20.120.194:38341)
0621 06:54:47.666171 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:47.666178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:47.666205 (+ 27us) server_negotiation.cc:407] Connection header received
0621 06:54:47.666265 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:47.666271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:47.666485 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:47.666626 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:47.667811 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:47.668809 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:47.670117 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:47.670690 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:47.670831 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:47.675115 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:47.675127 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:47.675140 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:47.675178 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:47.677712 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:47.678525 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:47.678541 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:47.678544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:47.678688 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:47.679112 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:47.679116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:47.679117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:47.679466 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:47.679752 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:47.680083 (+ 331us) server_negotiation.cc:299] Negotiation successful
0621 06:54:47.680378 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":318,"threads_started":1}
I20260621 06:54:47.682098 1162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:47.764602 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 1030
I20260621 06:54:47.780151 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 1173
Jun 21 06:54:47 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:54:47.796387 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:47.781216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.20.120.195:40397)
0621 06:54:47.781837 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:47.781845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:47.781875 (+ 30us) server_negotiation.cc:407] Connection header received
0621 06:54:47.781999 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:47.782005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:47.782215 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:47.782343 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:47.783517 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:47.784568 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:47.785877 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:47.786465 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:47.786613 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:47.791060 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:47.791071 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:47.791086 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:47.791126 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:47.794068 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:47.794676 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:47.794683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:47.794685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:47.794806 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:47.795136 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:47.795138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:47.795140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:47.795457 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:47.795671 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:47.795904 (+ 233us) server_negotiation.cc:299] Negotiation successful
0621 06:54:47.796095 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":353,"threads_started":1}
I20260621 06:54:47.797698 1296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:47.880824 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 1173
I20260621 06:54:47.896469 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 779
I20260621 06:54:47.897790 841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:48.003218 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 779
I20260621 06:54:48.027720 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:32873
--webserver_interface=127.20.120.254
--webserver_port=43379
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:32873
--ranger_config_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.120.212:54817/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.120.212:54817/kms with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:54:48.338203 1313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:48.338472 1313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:48.338550 1313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:48.344973 1313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:54:48.345062 1313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:48.345103 1313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:48.345145 1313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:54:48.345181 1313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:54:48.354466 1313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.120.212:54817/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:32873
--ranger_config_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=43379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:48.355628 1313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:48.356906 1313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:54:48.368145 1319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:48.368197 1318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:48.368225 1321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:48.369403 1313 server_base.cc:1034] running on GCE node
I20260621 06:54:48.370225 1313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:48.371621 1313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:48.372855 1313 hybrid_clock.cc:648] HybridClock initialized: now 1782024888372801 us; error 39 us; skew 500 ppm
I20260621 06:54:48.373224 1313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:54:48 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024888, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:48.376376 1313 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:54:48.377779 1313 webserver.cc:460] Webserver started at http://127.20.120.254:43379/ using document root <none> and password file <none>
I20260621 06:54:48.378458 1313 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:48.378609 1313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 06:54:48 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024888, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:54:48.464237 1313 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 06:54:48.467715 1329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:48.469122 1313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 06:54:48.469313 1313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "495dba44a6c643c19c67782f7a978808"
format_stamp: "Formatted at 2026-06-21 06:54:44 on dist-test-slave-gmf2"
server_key: "e3134fb6eeb68c94222993b2c86f21d9"
server_key_iv: "82315ee5f64fcb009fec79ccd6b8f8cd"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:48.470299 1313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:48.489697 1313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:48.493651 1313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:48.493983 1313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:48.512420 1313 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:32873
I20260621 06:54:48.512445 1380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:32873 every 8 connection(s)
I20260621 06:54:48.513805 1313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:54:48.520363 1381 sys_catalog.cc:263] Verifying existing consensus state
I20260621 06:54:48.521847 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 1313
I20260621 06:54:48.522900 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.120.212:54817/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:46031
--local_ip_for_outbound_sockets=127.20.120.193
--tserver_master_addrs=127.20.120.254:32873
--webserver_port=44119
--webserver_interface=127.20.120.193
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.120.212:54817/kms with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:54:48.523674 1381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808: Bootstrap starting.
I20260621 06:54:48.546628 1381 log.cc:824] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808: Log is configured to *not* fsync() on all Append() calls
I20260621 06:54:48.557821 1381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 06:54:48.558460 1381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808: Bootstrap complete.
I20260621 06:54:48.571584 1381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495dba44a6c643c19c67782f7a978808" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32873 } }
I20260621 06:54:48.572774 1381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495dba44a6c643c19c67782f7a978808, State: Initialized, Role: FOLLOWER
I20260621 06:54:48.573553 1381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495dba44a6c643c19c67782f7a978808" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32873 } }
I20260621 06:54:48.573812 1381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:54:48.573989 1381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:54:48.574174 1381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 1 FOLLOWER]: Advancing to term 2
I20260621 06:54:48.579721 1381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495dba44a6c643c19c67782f7a978808" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32873 } }
I20260621 06:54:48.580288 1381 leader_election.cc:302] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 495dba44a6c643c19c67782f7a978808; no voters:
I20260621 06:54:48.580889 1381 leader_election.cc:288] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 06:54:48.581032 1385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 06:54:48.582229 1385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [term 2 LEADER]: Becoming Leader. State: Replica: 495dba44a6c643c19c67782f7a978808, State: Running, Role: LEADER
I20260621 06:54:48.582746 1385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495dba44a6c643c19c67782f7a978808" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32873 } }
I20260621 06:54:48.583042 1381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:54:48.585444 1386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "495dba44a6c643c19c67782f7a978808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495dba44a6c643c19c67782f7a978808" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32873 } } }
I20260621 06:54:48.585464 1387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 495dba44a6c643c19c67782f7a978808. Latest consensus state: current_term: 2 leader_uuid: "495dba44a6c643c19c67782f7a978808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495dba44a6c643c19c67782f7a978808" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 32873 } } }
I20260621 06:54:48.585958 1387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [sys.catalog]: This master's current role is: LEADER
I20260621 06:54:48.585860 1386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808 [sys.catalog]: This master's current role is: LEADER
I20260621 06:54:48.587090 1392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:54:48.603093 1392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:54:48.604661 1392 catalog_manager.cc:1259] Loaded cluster ID: e3b4892d4ca74b4bab4391a80d580dce
I20260621 06:54:48.604768 1392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:54:48.609323 1392 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:54:48.611218 1392 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808: Loaded TSK: 0
I20260621 06:54:48.613250 1392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:54:48.619979 1381 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260621 06:54:49.004590 1383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:49.005005 1383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:49.005239 1383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:49.015259 1383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:49.015426 1383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:49.015676 1383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:54:49.028546 1383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.120.212:54817/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:46031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=44119
--tserver_master_addrs=127.20.120.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:49.030565 1383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:49.032608 1383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:54:49.080472 1418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:49.080749 1419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:49.093987 1383 server_base.cc:1034] running on GCE node
W20260621 06:54:49.094933 1421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:49.095934 1383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:49.097500 1383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:49.098778 1383 hybrid_clock.cc:648] HybridClock initialized: now 1782024889098656 us; error 126 us; skew 500 ppm
I20260621 06:54:49.099220 1383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:54:49 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:49.103299 1383 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:54:49.105193 1383 webserver.cc:460] Webserver started at http://127.20.120.193:44119/ using document root <none> and password file <none>
I20260621 06:54:49.106211 1383 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:49.106434 1383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 06:54:49 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:54:49.148324 1383 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.008s sys 0.008s
I20260621 06:54:49.152493 1429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:49.154572 1383 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260621 06:54:49.154778 1383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "209bcec7c7df4e199021233a7c13798e"
format_stamp: "Formatted at 2026-06-21 06:54:45 on dist-test-slave-gmf2"
server_key: "5deeedd9287b8d9b9223a2b66e4d0013"
server_key_iv: "5695bb4075bf31269615ce9c82695c34"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:49.156316 1383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:49.215260 1383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:49.220450 1383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:49.220774 1383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:49.226212 1383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:54:49.228281 1383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:54:49.228410 1383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:49.228536 1383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:54:49.228609 1383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:49.295555 1383 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:46031
I20260621 06:54:49.295599 1542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:46031 every 8 connection(s)
I20260621 06:54:49.297276 1383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:54:49.299436 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 1383
I20260621 06:54:49.300240 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.120.212:54817/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:38341
--local_ip_for_outbound_sockets=127.20.120.194
--tserver_master_addrs=127.20.120.254:32873
--webserver_port=34111
--webserver_interface=127.20.120.194
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.120.212:54817/kms with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 06:54:49 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:49.348461 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:49.304413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:32873 (local address 127.20.120.193:59901)
0621 06:54:49.320988 (+ 16575us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:49.321041 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:49.323411 (+ 2370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:49.326052 (+ 2641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:49.326066 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:49.326728 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:49.327569 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:49.327606 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:49.330521 (+ 2915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:49.330530 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:49.332934 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:49.332954 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:49.334526 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:49.334535 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:49.334915 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:49.335779 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:49.335849 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:49.338878 (+ 3029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:49.343404 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:49.343412 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:49.343456 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:49.344028 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:49.344583 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:49.344589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:49.344593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:49.344858 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:49.345980 (+ 1122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:49.345989 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:49.346360 (+ 371us) client_negotiation.cc:769] Sending connection context
0621 06:54:49.346632 (+ 272us) client_negotiation.cc:240] Negotiation successful
0621 06:54:49.347157 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16272,"thread_start_us":16080,"threads_started":1}
I20260621 06:54:49.351275 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:49.304385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:59901 (local address 127.20.120.254:32873)
0621 06:54:49.323918 (+ 19533us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:49.323939 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:49.323994 (+ 55us) server_negotiation.cc:407] Connection header received
0621 06:54:49.324525 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:49.324549 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:49.325116 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:49.327870 (+ 2754us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:49.328336 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:49.330330 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:49.333139 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:49.334254 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:49.334473 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:49.339191 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:49.339202 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:49.339222 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:49.339271 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:49.343240 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:49.344211 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:49.344219 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:49.344228 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:49.344418 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:49.345005 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:49.345010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:49.345013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:49.345650 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:49.345859 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:49.349528 (+ 3669us) server_negotiation.cc:299] Negotiation successful
0621 06:54:49.350119 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19215,"thread_start_us":20956,"threads_started":1}
I20260621 06:54:49.355136 1543 heartbeater.cc:347] Connected to a master server at 127.20.120.254:32873
I20260621 06:54:49.355485 1543 heartbeater.cc:464] Registering TS with master...
I20260621 06:54:49.356335 1543 heartbeater.cc:511] Master 127.20.120.254:32873 requested a full tablet report, sending...
W20260621 06:54:49.359525 1543 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:32873 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 06:54:49.359730 1543 heartbeater.cc:464] Registering TS with master...
I20260621 06:54:49.360014 1543 heartbeater.cc:511] Master 127.20.120.254:32873 requested a full tablet report, sending...
I20260621 06:54:49.361213 1543 heartbeater.cc:464] Registering TS with master...
I20260621 06:54:49.361608 1543 heartbeater.cc:511] Master 127.20.120.254:32873 requested a full tablet report, sending...
W20260621 06:54:49.362722 1543 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 06:54:49.728686 1547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:49.728998 1547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:49.729153 1547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:49.739185 1547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:49.739331 1547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:49.739574 1547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:54:49.753109 1547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.120.212:54817/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:38341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=34111
--tserver_master_addrs=127.20.120.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:49.754541 1547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:49.755875 1547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:54:49.772886 1561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:49.774760 1562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:49.778893 1547 server_base.cc:1034] running on GCE node
W20260621 06:54:49.779289 1564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:49.780022 1547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:49.790220 1547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:49.791465 1547 hybrid_clock.cc:648] HybridClock initialized: now 1782024889791393 us; error 63 us; skew 500 ppm
I20260621 06:54:49.791811 1547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:54:49 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:49.795995 1547 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:54:49.801932 1547 webserver.cc:460] Webserver started at http://127.20.120.194:34111/ using document root <none> and password file <none>
I20260621 06:54:49.802852 1547 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:49.803009 1547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 06:54:49 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:54:49.842156 1547 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.005s sys 0.012s
I20260621 06:54:49.846022 1574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:49.847983 1547 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260621 06:54:49.848170 1547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "e1b05c36cdb847fe8e59b312bb579812"
format_stamp: "Formatted at 2026-06-21 06:54:46 on dist-test-slave-gmf2"
server_key: "cfa4ac12d3d4daa956ed9248cc306fa8"
server_key_iv: "b6d530d46bccff265614e3fdc5c46e80"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:49.849627 1547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 21 06:54:49 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782024889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:49.887473 1547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:49.892506 1547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:49.892799 1547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:49.894933 1547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:54:49.896760 1547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:54:49.896873 1547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:49.896965 1547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:54:49.897027 1547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:49.993643 1547 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:38341
I20260621 06:54:49.993816 1687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:38341 every 8 connection(s)
I20260621 06:54:49.995489 1547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:54:49.997671 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 1547
I20260621 06:54:49.998718 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.120.212:54817/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:40397
--local_ip_for_outbound_sockets=127.20.120.195
--tserver_master_addrs=127.20.120.254:32873
--webserver_port=33041
--webserver_interface=127.20.120.195
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.120.212:54817/kms with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 06:54:50 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:50.054785 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:49.999866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:32873 (local address 127.20.120.194:47047)
0621 06:54:50.027448 (+ 27582us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:50.027520 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:50.030681 (+ 3161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:50.031993 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:50.032007 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:50.033347 (+ 1340us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:50.034363 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:50.034391 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.036048 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.036054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:50.040330 (+ 4276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:50.040363 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.041783 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.041791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:50.042165 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:50.043054 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:50.043087 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:50.046036 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:50.050099 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:50.050108 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:50.050127 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:50.050706 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:50.051186 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:50.051191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:50.051195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:50.051436 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:50.052147 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:50.052156 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:50.052572 (+ 416us) client_negotiation.cc:769] Sending connection context
0621 06:54:50.052867 (+ 295us) client_negotiation.cc:240] Negotiation successful
0621 06:54:50.053446 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27167,"thread_start_us":28469,"threads_started":1}
I20260621 06:54:50.056499 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.000657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:47047 (local address 127.20.120.254:32873)
0621 06:54:50.025038 (+ 24381us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:50.025045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:50.028420 (+ 3375us) server_negotiation.cc:407] Connection header received
0621 06:54:50.031229 (+ 2809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:50.031235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:50.031349 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:50.031467 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:50.034604 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.035807 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.040556 (+ 4749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.041272 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.041378 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:50.046357 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:50.046364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:50.046369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:50.046419 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:50.049888 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:50.050878 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:50.050885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:50.050889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:50.051064 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:50.051561 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:50.051567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:50.051570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:50.051941 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:50.052074 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:50.055924 (+ 3850us) server_negotiation.cc:299] Negotiation successful
0621 06:54:50.056161 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24209,"thread_start_us":24400,"threads_started":1}
I20260621 06:54:50.058323 1688 heartbeater.cc:347] Connected to a master server at 127.20.120.254:32873
I20260621 06:54:50.058641 1688 heartbeater.cc:464] Registering TS with master...
I20260621 06:54:50.059471 1688 heartbeater.cc:511] Master 127.20.120.254:32873 requested a full tablet report, sending...
I20260621 06:54:50.061470 1346 ts_manager.cc:194] Registered new tserver with Master: e1b05c36cdb847fe8e59b312bb579812 (127.20.120.194:38341)
I20260621 06:54:50.063642 1346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:47047
I20260621 06:54:50.363497 1543 heartbeater.cc:464] Registering TS with master...
I20260621 06:54:50.363916 1543 heartbeater.cc:511] Master 127.20.120.254:32873 requested a full tablet report, sending...
I20260621 06:54:50.365424 1346 ts_manager.cc:194] Registered new tserver with Master: 209bcec7c7df4e199021233a7c13798e (127.20.120.193:46031)
I20260621 06:54:50.366746 1346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:59901
Jun 21 06:54:50 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
W20260621 06:54:50.449630 1692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:50.450101 1692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:50.450285 1692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:50.458139 1692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:50.458269 1692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:50.458454 1692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:54:50.468720 1692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.120.212:54817/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:40397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=33041
--tserver_master_addrs=127.20.120.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:50.470224 1692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:50.471529 1692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:54:50.488188 1698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:50.490069 1697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:50.490489 1700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:50.491201 1692 server_base.cc:1034] running on GCE node
I20260621 06:54:50.491839 1692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:50.492822 1692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:50.494014 1692 hybrid_clock.cc:648] HybridClock initialized: now 1782024890493949 us; error 53 us; skew 500 ppm
I20260621 06:54:50.494342 1692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:54:50 dist-test-slave-gmf2 krb5kdc[335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024890, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:50.497373 1692 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:54:50.498947 1692 webserver.cc:460] Webserver started at http://127.20.120.195:33041/ using document root <none> and password file <none>
I20260621 06:54:50.499656 1692 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:50.499799 1692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 06:54:50 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024890, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:54:50.519472 1692 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 06:54:50.522537 1708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:50.524209 1692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 06:54:50.524434 1692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "97f432eeeb1f4e5a9e4479284b48fde3"
format_stamp: "Formatted at 2026-06-21 06:54:47 on dist-test-slave-gmf2"
server_key: "6647d7190922687a215ca726a2d8c426"
server_key_iv: "4de647b4198db8f011dccc6f1e308b9a"
server_key_version: "kuduclusterkey@0"
I20260621 06:54:50.525854 1692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:50.564602 1692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:50.569784 1692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:50.570191 1692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:50.571409 1692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:54:50.573218 1692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:54:50.573401 1692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:50.573573 1692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:54:50.573709 1692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:50.626317 1692 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:40397
I20260621 06:54:50.626387 1820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:40397 every 8 connection(s)
I20260621 06:54:50.628271 1692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:54:50.638481 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 1692
Jun 21 06:54:50 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024890, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:50.681597 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.640309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42568 (local address 127.20.120.254:32873)
0621 06:54:50.653484 (+ 13175us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:50.653494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:50.653524 (+ 30us) server_negotiation.cc:407] Connection header received
0621 06:54:50.653627 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:50.653637 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:50.653761 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:50.653972 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:50.656438 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.657553 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.659512 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.660125 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.660268 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:50.664398 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:50.664403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:50.664407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:50.664450 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:50.673467 (+ 9017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:50.674530 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:50.674536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:50.674539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:50.674722 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:50.675203 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:50.675208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:50.675211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:50.678082 (+ 2871us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:50.680832 (+ 2750us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:50.680973 (+ 141us) server_negotiation.cc:299] Negotiation successful
0621 06:54:50.681220 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13002,"thread_start_us":12808,"threads_started":1}
I20260621 06:54:50.684551 1825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.639930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:48077 (local address 127.20.120.254:32873)
0621 06:54:50.647078 (+ 7148us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:50.647084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:50.654986 (+ 7902us) server_negotiation.cc:407] Connection header received
0621 06:54:50.659172 (+ 4186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:50.659177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:50.659255 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:50.659326 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:50.662927 (+ 3601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.664184 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.667556 (+ 3372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.668328 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.668459 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:50.672642 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:50.672648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:50.672653 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:50.672692 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:50.676038 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:50.676895 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:50.676900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:50.676903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:50.677040 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:50.677510 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:50.677515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:50.677518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:50.677811 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:50.677968 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:50.683966 (+ 5998us) server_negotiation.cc:299] Negotiation successful
0621 06:54:50.684189 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7018,"thread_start_us":7282,"threads_started":1}
I20260621 06:54:50.685338 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.637732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:32873 (local address 127.20.120.195:48077)
0621 06:54:50.654802 (+ 17070us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:50.654875 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:50.658813 (+ 3938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:50.660903 (+ 2090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:50.660916 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:50.661493 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:50.662622 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:50.662730 (+ 108us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.665212 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.665218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:50.667369 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:50.667387 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.668554 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.668563 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:50.668892 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:50.669614 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:50.669645 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:50.672328 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:50.676190 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:50.676199 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:50.676216 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:50.676745 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:50.677158 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:50.677163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:50.677166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:50.677378 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:50.678037 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:50.678045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:50.678448 (+ 403us) client_negotiation.cc:769] Sending connection context
0621 06:54:50.678735 (+ 287us) client_negotiation.cc:240] Negotiation successful
0621 06:54:50.679275 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16764,"thread_start_us":16532,"threads_started":1}
I20260621 06:54:50.687369 1821 heartbeater.cc:347] Connected to a master server at 127.20.120.254:32873
I20260621 06:54:50.687721 1821 heartbeater.cc:464] Registering TS with master...
I20260621 06:54:50.688680 1821 heartbeater.cc:511] Master 127.20.120.254:32873 requested a full tablet report, sending...
I20260621 06:54:50.690483 1346 ts_manager.cc:194] Registered new tserver with Master: 97f432eeeb1f4e5a9e4479284b48fde3 (127.20.120.195:40397)
I20260621 06:54:50.691535 1346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:48077
Jun 21 06:54:50 dist-test-slave-gmf2 krb5kdc[335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for HTTP/127.20.120.212@KRBTEST.COM
I20260621 06:54:50.703580 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:54:50.758248 1825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.730506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.20.120.254:32873)
0621 06:54:50.730725 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:50.730732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:50.740991 (+ 10259us) server_negotiation.cc:407] Connection header received
0621 06:54:50.741476 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:50.741483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:50.741577 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:50.741686 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:50.743642 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.744715 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.746754 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.747438 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.747535 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:50.751243 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:50.751250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:50.751255 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:50.751311 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:50.754980 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:50.755834 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:50.755840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:50.755844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:50.755994 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:50.756456 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:50.756461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:50.756465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:50.756828 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:50.757086 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:50.757658 (+ 572us) server_negotiation.cc:299] Negotiation successful
0621 06:54:50.757884 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260621 06:54:50.765488 1346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:54:50.776871 1346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:54:50.884369 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.806357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:40397 (local address 127.0.0.1:45524)
0621 06:54:50.865310 (+ 58953us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:50.865332 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:50.865466 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:50.866675 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:50.866682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:50.866712 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:54:50.867104 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:50.867117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.871072 (+ 3955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.871078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:50.873482 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:50.873499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.883289 (+ 9790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.883296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:50.883522 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:50.883587 (+ 65us) client_negotiation.cc:769] Sending connection context
0621 06:54:50.883742 (+ 155us) client_negotiation.cc:240] Negotiation successful
0621 06:54:50.884012 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58835,"thread_start_us":59184,"threads_started":1}
I20260621 06:54:50.886112 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.803558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:46031 (local address 127.0.0.1:56280)
0621 06:54:50.862185 (+ 58627us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:50.862244 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:50.862488 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:50.868558 (+ 6070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:50.868563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:50.868590 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:54:50.868944 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:50.868971 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.879364 (+ 10393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.879370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:50.883034 (+ 3664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:50.883050 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.885286 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.885292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:50.885463 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:50.885496 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 06:54:50.885581 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 06:54:50.885718 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58274,"thread_start_us":72828,"threads_started":1}
I20260621 06:54:50.887153 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.806848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.20.120.195:40397)
0621 06:54:50.861516 (+ 54668us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:50.861527 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:50.865739 (+ 4212us) server_negotiation.cc:407] Connection header received
0621 06:54:50.865849 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:50.865857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:50.866149 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:50.866343 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:54:50.869184 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.870881 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.874930 (+ 4049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.878857 (+ 3927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.879079 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:50.879276 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:50.885265 (+ 5989us) server_negotiation.cc:299] Negotiation successful
0621 06:54:50.885461 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54449,"thread_start_us":56475,"threads_started":1}
I20260621 06:54:50.888207 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.803796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.20.120.193:46031)
0621 06:54:50.867691 (+ 63895us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:50.867701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:50.867731 (+ 30us) server_negotiation.cc:407] Connection header received
0621 06:54:50.867825 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:50.867833 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:50.868096 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:50.868294 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:54:50.873710 (+ 5416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.878161 (+ 4451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.883194 (+ 5033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.884750 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.884967 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:50.885144 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:50.887687 (+ 2543us) server_negotiation.cc:299] Negotiation successful
0621 06:54:50.887876 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63706,"thread_start_us":63612,"threads_started":1}
I20260621 06:54:50.892160 1756 tablet_service.cc:1467] Processing CreateTablet for tablet 7fb9208fb4804179b68b77c083338f1f (DEFAULT_TABLE table=test-table [id=2c3721b133764efd8776ef41bcb5b1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:54:50.892154 1478 tablet_service.cc:1467] Processing CreateTablet for tablet 7fb9208fb4804179b68b77c083338f1f (DEFAULT_TABLE table=test-table [id=2c3721b133764efd8776ef41bcb5b1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:54:50.893674 1756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fb9208fb4804179b68b77c083338f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:54:50.894184 1478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fb9208fb4804179b68b77c083338f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:54:50.905577 1842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.823203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.20.120.194:38341)
0621 06:54:50.863104 (+ 39901us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:50.863115 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:50.878428 (+ 15313us) server_negotiation.cc:407] Connection header received
0621 06:54:50.878744 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:50.878752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:50.879036 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:50.879220 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:54:50.880174 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.881691 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.902213 (+ 20522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.903863 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.904024 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:50.904216 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:50.905046 (+ 830us) server_negotiation.cc:299] Negotiation successful
0621 06:54:50.905241 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39720,"thread_start_us":39926,"threads_started":1}
I20260621 06:54:50.906351 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.822648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:38341 (local address 127.0.0.1:54216)
0621 06:54:50.876338 (+ 53690us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:50.878317 (+ 1979us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:50.878548 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:50.879519 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:50.879524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:50.879550 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:54:50.879965 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:50.879980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.881882 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.881887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:50.887495 (+ 5608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:50.887510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:50.904550 (+ 17040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:50.904557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:50.904779 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:50.904814 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 06:54:50.905777 (+ 963us) client_negotiation.cc:240] Negotiation successful
0621 06:54:50.905979 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53555,"mutex_wait_us":1,"thread_start_us":40842,"threads_started":1}
I20260621 06:54:50.910449 1623 tablet_service.cc:1467] Processing CreateTablet for tablet 7fb9208fb4804179b68b77c083338f1f (DEFAULT_TABLE table=test-table [id=2c3721b133764efd8776ef41bcb5b1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:54:50.911863 1623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fb9208fb4804179b68b77c083338f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:54:50.918570 1843 tablet_bootstrap.cc:492] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e: Bootstrap starting.
I20260621 06:54:50.920379 1844 tablet_bootstrap.cc:492] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3: Bootstrap starting.
I20260621 06:54:50.924021 1844 tablet_bootstrap.cc:654] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3: Neither blocks nor log segments found. Creating new log.
I20260621 06:54:50.925323 1843 tablet_bootstrap.cc:654] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e: Neither blocks nor log segments found. Creating new log.
I20260621 06:54:50.925395 1844 log.cc:824] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3: Log is configured to *not* fsync() on all Append() calls
I20260621 06:54:50.926551 1843 log.cc:824] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e: Log is configured to *not* fsync() on all Append() calls
I20260621 06:54:50.942871 1843 tablet_bootstrap.cc:492] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e: No bootstrap required, opened a new log
I20260621 06:54:50.944458 1843 ts_tablet_manager.cc:1397] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.013s
I20260621 06:54:50.948185 1844 tablet_bootstrap.cc:492] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3: No bootstrap required, opened a new log
I20260621 06:54:50.948437 1844 ts_tablet_manager.cc:1397] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.007s
I20260621 06:54:50.954821 1843 raft_consensus.cc:348] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209bcec7c7df4e199021233a7c13798e" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 46031 } } peers { permanent_uuid: "97f432eeeb1f4e5a9e4479284b48fde3" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40397 } } peers { permanent_uuid: "e1b05c36cdb847fe8e59b312bb579812" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38341 } }
I20260621 06:54:50.955288 1843 raft_consensus.cc:374] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:54:50.955387 1843 raft_consensus.cc:729] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209bcec7c7df4e199021233a7c13798e, State: Initialized, Role: FOLLOWER
I20260621 06:54:50.956439 1848 tablet_bootstrap.cc:492] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812: Bootstrap starting.
I20260621 06:54:50.960196 1844 raft_consensus.cc:348] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209bcec7c7df4e199021233a7c13798e" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 46031 } } peers { permanent_uuid: "97f432eeeb1f4e5a9e4479284b48fde3" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40397 } } peers { permanent_uuid: "e1b05c36cdb847fe8e59b312bb579812" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38341 } }
I20260621 06:54:50.960758 1844 raft_consensus.cc:374] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:54:50.960896 1844 raft_consensus.cc:729] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f432eeeb1f4e5a9e4479284b48fde3, State: Initialized, Role: FOLLOWER
I20260621 06:54:50.960729 1843 consensus_queue.cc:260] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209bcec7c7df4e199021233a7c13798e" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 46031 } } peers { permanent_uuid: "97f432eeeb1f4e5a9e4479284b48fde3" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40397 } } peers { permanent_uuid: "e1b05c36cdb847fe8e59b312bb579812" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38341 } }
I20260621 06:54:50.961964 1844 consensus_queue.cc:260] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209bcec7c7df4e199021233a7c13798e" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 46031 } } peers { permanent_uuid: "97f432eeeb1f4e5a9e4479284b48fde3" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40397 } } peers { permanent_uuid: "e1b05c36cdb847fe8e59b312bb579812" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38341 } }
I20260621 06:54:50.963006 1848 tablet_bootstrap.cc:654] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812: Neither blocks nor log segments found. Creating new log.
I20260621 06:54:50.964828 1848 log.cc:824] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812: Log is configured to *not* fsync() on all Append() calls
I20260621 06:54:50.969527 1848 tablet_bootstrap.cc:492] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812: No bootstrap required, opened a new log
I20260621 06:54:50.969825 1848 ts_tablet_manager.cc:1397] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20260621 06:54:50.971269 1843 ts_tablet_manager.cc:1428] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e: Time spent starting tablet: real 0.027s user 0.007s sys 0.008s
I20260621 06:54:50.971714 1543 heartbeater.cc:503] Master 127.20.120.254:32873 was elected leader, sending a full tablet report...
I20260621 06:54:50.980938 1848 raft_consensus.cc:348] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209bcec7c7df4e199021233a7c13798e" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 46031 } } peers { permanent_uuid: "97f432eeeb1f4e5a9e4479284b48fde3" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40397 } } peers { permanent_uuid: "e1b05c36cdb847fe8e59b312bb579812" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38341 } }
I20260621 06:54:50.981452 1848 raft_consensus.cc:374] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:54:50.981572 1848 raft_consensus.cc:729] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b05c36cdb847fe8e59b312bb579812, State: Initialized, Role: FOLLOWER
I20260621 06:54:50.982353 1848 consensus_queue.cc:260] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209bcec7c7df4e199021233a7c13798e" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 46031 } } peers { permanent_uuid: "97f432eeeb1f4e5a9e4479284b48fde3" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40397 } } peers { permanent_uuid: "e1b05c36cdb847fe8e59b312bb579812" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38341 } }
I20260621 06:54:50.984521 1850 raft_consensus.cc:482] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:54:50.984789 1850 raft_consensus.cc:504] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209bcec7c7df4e199021233a7c13798e" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 46031 } } peers { permanent_uuid: "97f432eeeb1f4e5a9e4479284b48fde3" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40397 } } peers { permanent_uuid: "e1b05c36cdb847fe8e59b312bb579812" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38341 } }
I20260621 06:54:50.990899 1848 ts_tablet_manager.cc:1428] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812: Time spent starting tablet: real 0.021s user 0.017s sys 0.002s
I20260621 06:54:50.991262 1688 heartbeater.cc:503] Master 127.20.120.254:32873 was elected leader, sending a full tablet report...
I20260621 06:54:51.000965 1850 leader_election.cc:288] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f432eeeb1f4e5a9e4479284b48fde3 (127.20.120.195:40397), e1b05c36cdb847fe8e59b312bb579812 (127.20.120.194:38341)
I20260621 06:54:51.002054 1821 heartbeater.cc:503] Master 127.20.120.254:32873 was elected leader, sending a full tablet report...
I20260621 06:54:51.002895 1844 ts_tablet_manager.cc:1428] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3: Time spent starting tablet: real 0.054s user 0.026s sys 0.023s
I20260621 06:54:51.019898 1842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.001338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:45711 (local address 127.20.120.194:38341)
0621 06:54:51.001564 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:51.001570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:51.013305 (+ 11735us) server_negotiation.cc:407] Connection header received
0621 06:54:51.013540 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:51.013548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:51.013642 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:51.013751 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:54:51.015246 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.016029 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.017951 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.018904 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.019063 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:51.019133 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:51.019473 (+ 340us) server_negotiation.cc:299] Negotiation successful
0621 06:54:51.019610 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 06:54:51.020011 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.001417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:38341 (local address 127.20.120.193:45711)
0621 06:54:51.013218 (+ 11801us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:51.013234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:51.013379 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:51.014623 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:51.014632 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:51.014664 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:54:51.015056 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:51.015069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.016167 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.016172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:51.017762 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:51.017774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.019123 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.019130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:51.019292 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:51.019325 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 06:54:51.019437 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 06:54:51.019564 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11695,"mutex_wait_us":76,"thread_start_us":12023,"threads_started":1}
I20260621 06:54:51.021117 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.988297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:33917 (local address 127.20.120.195:40397)
0621 06:54:50.988573 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:50.988582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:51.013317 (+ 24735us) server_negotiation.cc:407] Connection header received
0621 06:54:51.013968 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:51.013973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:51.014051 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:51.014134 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:54:51.014912 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.016002 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.018592 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.019651 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.019815 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:51.019880 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:51.020759 (+ 879us) server_negotiation.cc:299] Negotiation successful
0621 06:54:51.020892 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260621 06:54:51.021234 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:50.987590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:40397 (local address 127.20.120.193:33917)
0621 06:54:51.013165 (+ 25575us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:51.013186 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:51.013359 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:51.014298 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:51.014303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:51.014329 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:54:51.014724 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:51.014743 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.016175 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.016180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:51.018429 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:51.018443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.020075 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.020081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:51.020253 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:51.020284 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 06:54:51.020731 (+ 447us) client_negotiation.cc:240] Negotiation successful
0621 06:54:51.020872 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25404,"thread_start_us":25103,"threads_started":1}
I20260621 06:54:51.021180 1643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb9208fb4804179b68b77c083338f1f" candidate_uuid: "209bcec7c7df4e199021233a7c13798e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1b05c36cdb847fe8e59b312bb579812" is_pre_election: true
I20260621 06:54:51.021798 1643 raft_consensus.cc:2454] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209bcec7c7df4e199021233a7c13798e in term 0.
I20260621 06:54:51.022619 1432 leader_election.cc:302] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209bcec7c7df4e199021233a7c13798e, e1b05c36cdb847fe8e59b312bb579812; no voters:
I20260621 06:54:51.022604 1776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb9208fb4804179b68b77c083338f1f" candidate_uuid: "209bcec7c7df4e199021233a7c13798e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f432eeeb1f4e5a9e4479284b48fde3" is_pre_election: true
I20260621 06:54:51.023046 1776 raft_consensus.cc:2454] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209bcec7c7df4e199021233a7c13798e in term 0.
I20260621 06:54:51.023066 1850 raft_consensus.cc:2781] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:54:51.023209 1850 raft_consensus.cc:482] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:54:51.023308 1850 raft_consensus.cc:3037] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:54:51.026618 1850 raft_consensus.cc:504] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209bcec7c7df4e199021233a7c13798e" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 46031 } } peers { permanent_uuid: "97f432eeeb1f4e5a9e4479284b48fde3" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40397 } } peers { permanent_uuid: "e1b05c36cdb847fe8e59b312bb579812" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38341 } }
I20260621 06:54:51.027431 1850 leader_election.cc:288] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [CANDIDATE]: Term 1 election: Requested vote from peers 97f432eeeb1f4e5a9e4479284b48fde3 (127.20.120.195:40397), e1b05c36cdb847fe8e59b312bb579812 (127.20.120.194:38341)
I20260621 06:54:51.027736 1776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb9208fb4804179b68b77c083338f1f" candidate_uuid: "209bcec7c7df4e199021233a7c13798e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f432eeeb1f4e5a9e4479284b48fde3"
I20260621 06:54:51.027957 1776 raft_consensus.cc:3037] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:54:51.027930 1643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb9208fb4804179b68b77c083338f1f" candidate_uuid: "209bcec7c7df4e199021233a7c13798e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1b05c36cdb847fe8e59b312bb579812"
I20260621 06:54:51.028119 1643 raft_consensus.cc:3037] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:54:51.030180 1776 raft_consensus.cc:2454] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209bcec7c7df4e199021233a7c13798e in term 1.
I20260621 06:54:51.030658 1433 leader_election.cc:302] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209bcec7c7df4e199021233a7c13798e, 97f432eeeb1f4e5a9e4479284b48fde3; no voters:
I20260621 06:54:51.030964 1850 raft_consensus.cc:2781] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:54:51.031262 1643 raft_consensus.cc:2454] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209bcec7c7df4e199021233a7c13798e in term 1.
I20260621 06:54:51.032064 1850 raft_consensus.cc:686] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 1 LEADER]: Becoming Leader. State: Replica: 209bcec7c7df4e199021233a7c13798e, State: Running, Role: LEADER
I20260621 06:54:51.032565 1850 consensus_queue.cc:237] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209bcec7c7df4e199021233a7c13798e" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 46031 } } peers { permanent_uuid: "97f432eeeb1f4e5a9e4479284b48fde3" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40397 } } peers { permanent_uuid: "e1b05c36cdb847fe8e59b312bb579812" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38341 } }
I20260621 06:54:51.037424 1346 catalog_manager.cc:5495] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e reported cstate change: term changed from 0 to 1, leader changed from <none> to 209bcec7c7df4e199021233a7c13798e (127.20.120.193). New cstate: current_term: 1 leader_uuid: "209bcec7c7df4e199021233a7c13798e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209bcec7c7df4e199021233a7c13798e" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 46031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97f432eeeb1f4e5a9e4479284b48fde3" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1b05c36cdb847fe8e59b312bb579812" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38341 } health_report { overall_health: UNKNOWN } } }
W20260621 06:54:51.046268 1689 tablet.cc:2369] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:54:51.054438 1544 tablet.cc:2369] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:54:51.083834 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.078511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.20.120.193:46031)
0621 06:54:51.078691 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:51.078699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:51.078753 (+ 54us) server_negotiation.cc:407] Connection header received
0621 06:54:51.079031 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:51.079035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:51.079120 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:51.079270 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:54:51.079661 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.080341 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.081666 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.082105 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.082255 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:51.082746 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:54:51.083242 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:54:51.083413 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:51.083468 (+ 55us) server_negotiation.cc:299] Negotiation successful
0621 06:54:51.083601 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 06:54:51.091841 1643 raft_consensus.cc:1261] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812 [term 1 FOLLOWER]: Refusing update from remote peer 209bcec7c7df4e199021233a7c13798e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:54:51.092001 1776 raft_consensus.cc:1261] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3 [term 1 FOLLOWER]: Refusing update from remote peer 209bcec7c7df4e199021233a7c13798e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:54:51.092885 1861 consensus_queue.cc:1035] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1b05c36cdb847fe8e59b312bb579812" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:54:51.093266 1850 consensus_queue.cc:1035] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f432eeeb1f4e5a9e4479284b48fde3" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:54:51.111176 1864 mvcc.cc:205] Tried to move back new op lower bound from 7299173953902661632 to 7299173953674035200. Current Snapshot: MvccSnapshot[applied={T|T < 7299173953902661632}]
I20260621 06:54:51.126063 1866 mvcc.cc:205] Tried to move back new op lower bound from 7299173953902661632 to 7299173953674035200. Current Snapshot: MvccSnapshot[applied={T|T < 7299173953902661632 or (T in {7299173953902661632})}]
I20260621 06:54:51.127889 1346 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:54:51.128151 1346 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:54:51.132216 1346 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 495dba44a6c643c19c67782f7a978808: Sending DeleteTablet for 3 replicas of tablet 7fb9208fb4804179b68b77c083338f1f
I20260621 06:54:51.133494 1623 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fb9208fb4804179b68b77c083338f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:54:51 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:54216
I20260621 06:54:51.133769 1756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fb9208fb4804179b68b77c083338f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:54:51 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:45524
I20260621 06:54:51.134790 1478 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fb9208fb4804179b68b77c083338f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:54:51 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:56280
I20260621 06:54:51.143414 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 1383
I20260621 06:54:51.162194 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.145157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.20.120.193:46031)
0621 06:54:51.145349 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:51.145354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:51.145374 (+ 20us) server_negotiation.cc:407] Connection header received
0621 06:54:51.145430 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:51.145435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:51.145543 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:51.145648 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:51.148095 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.149317 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.151235 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.151897 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.152026 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:51.155156 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:51.155172 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:51.155196 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:51.155247 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:51.159341 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:51.160105 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:51.160113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:51.160117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:51.160294 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:51.160638 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:51.160642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:51.160646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:51.161035 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:51.161339 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:51.161561 (+ 222us) server_negotiation.cc:299] Negotiation successful
0621 06:54:51.161816 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 06:54:51.163170 1538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:51.168183 1874 tablet_replica.cc:330] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3: stopping tablet replica
I20260621 06:54:51.168627 1874 raft_consensus.cc:2229] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:54:51.169138 1874 raft_consensus.cc:2258] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:54:51.170543 1873 tablet_replica.cc:330] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812: stopping tablet replica
I20260621 06:54:51.171540 1873 raft_consensus.cc:2229] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:54:51.172358 1874 ts_tablet_manager.cc:1905] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:54:51.172541 1873 raft_consensus.cc:2258] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:54:51.174381 1873 ts_tablet_manager.cc:1905] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:54:51.179661 1874 ts_tablet_manager.cc:1918] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:54:51.179842 1874 log.cc:1196] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/7fb9208fb4804179b68b77c083338f1f
I20260621 06:54:51.180401 1874 ts_tablet_manager.cc:1939] T 7fb9208fb4804179b68b77c083338f1f P 97f432eeeb1f4e5a9e4479284b48fde3: Deleting consensus metadata
I20260621 06:54:51.180886 1873 ts_tablet_manager.cc:1918] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:54:51.181072 1873 log.cc:1196] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/7fb9208fb4804179b68b77c083338f1f
I20260621 06:54:51.181578 1873 ts_tablet_manager.cc:1939] T 7fb9208fb4804179b68b77c083338f1f P e1b05c36cdb847fe8e59b312bb579812: Deleting consensus metadata
I20260621 06:54:51.182327 1333 catalog_manager.cc:4841] TS 97f432eeeb1f4e5a9e4479284b48fde3 (127.20.120.195:40397): tablet 7fb9208fb4804179b68b77c083338f1f (table test-table [id=2c3721b133764efd8776ef41bcb5b1ae]) successfully deleted
I20260621 06:54:51.183804 1332 catalog_manager.cc:4841] TS e1b05c36cdb847fe8e59b312bb579812 (127.20.120.194:38341): tablet 7fb9208fb4804179b68b77c083338f1f (table test-table [id=2c3721b133764efd8776ef41bcb5b1ae]) successfully deleted
I20260621 06:54:51.275190 1875 tablet_replica.cc:330] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e: stopping tablet replica
I20260621 06:54:51.275723 1875 raft_consensus.cc:2229] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:54:51.276453 1875 raft_consensus.cc:2258] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:54:51.278748 1875 ts_tablet_manager.cc:1905] T 7fb9208fb4804179b68b77c083338f1f P 209bcec7c7df4e199021233a7c13798e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:54:51.281957 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 1383
W20260621 06:54:51.305434 1330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.120.193:46031 (error 108)
I20260621 06:54:51.305598 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 1547
I20260621 06:54:51.307415 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.306928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:46031 (local address 127.0.0.1:56294)
0621 06:54:51.307118 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:51.307240 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260621 06:54:51.307706 1330 catalog_manager.cc:4584] TS 209bcec7c7df4e199021233a7c13798e (127.20.120.193:46031): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7fb9208fb4804179b68b77c083338f1f: Network error: Client connection negotiation failed: client connection to 127.20.120.193:46031: connect: Connection refused (error 111)
I20260621 06:54:51.320812 1842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.306929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.20.120.194:38341)
0621 06:54:51.307133 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:51.307138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:51.307157 (+ 19us) server_negotiation.cc:407] Connection header received
0621 06:54:51.307220 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:51.307225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:51.307328 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:51.307440 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:51.309199 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.309960 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.311737 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.312153 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.312338 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:51.315029 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:51.315042 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:51.315057 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:51.315093 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:51.317995 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:51.318684 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:51.318704 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:51.318707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:51.318835 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:51.319231 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:51.319237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:51.319241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:51.319587 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:51.319917 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:51.320212 (+ 295us) server_negotiation.cc:299] Negotiation successful
0621 06:54:51.320465 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260621 06:54:51.321597 1683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:51.361335 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.360849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:46031 (local address 127.0.0.1:56304)
0621 06:54:51.361064 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:51.361178 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260621 06:54:51.362438 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.362087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:46031 (local address 127.0.0.1:56312)
0621 06:54:51.362236 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:51.362326 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260621 06:54:51.411806 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 1547
I20260621 06:54:51.423182 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.422600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:46031 (local address 127.0.0.1:56322)
0621 06:54:51.422845 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:51.422975 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260621 06:54:51.424335 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.423865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:46031 (local address 127.0.0.1:56324)
0621 06:54:51.424061 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:51.424160 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 06:54:51.428494 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 1692
I20260621 06:54:51.443806 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.429252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.20.120.195:40397)
0621 06:54:51.429411 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:51.429433 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:51.429452 (+ 19us) server_negotiation.cc:407] Connection header received
0621 06:54:51.429608 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:51.429612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:51.429673 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:51.429753 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:51.431216 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.431968 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.433266 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:51.433655 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:51.433748 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:51.436393 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:51.436416 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:51.436440 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:51.436488 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:51.440160 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:51.441124 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:51.441134 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:51.441138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:51.441299 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:51.441721 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:51.441726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:51.441729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:51.442180 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:51.442474 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:51.443094 (+ 620us) server_negotiation.cc:299] Negotiation successful
0621 06:54:51.443380 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 06:54:51.444481 1816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:51.504819 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.504349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:46031 (local address 127.0.0.1:56336)
0621 06:54:51.504548 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:51.504629 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260621 06:54:51.505829 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.505473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:46031 (local address 127.0.0.1:56340)
0621 06:54:51.505660 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:51.505726 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260621 06:54:51.567138 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 1692
I20260621 06:54:51.583439 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 1313
I20260621 06:54:51.584578 1376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:51.693161 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:51.692434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:46031 (local address 127.0.0.1:56350)
0621 06:54:51.692732 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:51.692863 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260621 06:54:51.693394 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 1313
I20260621 06:54:51.717710 20963 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 06:54:52.053434 20963 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 06:54:52.053661 20963 mini_ranger.cc:66] Stopping Ranger...
I20260621 06:54:52.398839 20963 mini_ranger.cc:68] Stopped Ranger
2026-06-21 06:54:52.399 UTC [365] LOG: received smart shutdown request
2026-06-21 06:54:52.402 UTC [365] LOG: background worker "logical replication launcher" (PID 374) exited with exit code 1
2026-06-21 06:54:52.402 UTC [369] LOG: shutting down
2026-06-21 06:54:52.549 UTC [365] LOG: database system is shut down
2026-06-21T06:54:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50052 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:54:52 dist-test-slave-gmf2 krb5kdc[1889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:54:52 dist-test-slave-gmf2 krb5kdc[1889](info): set up 2 sockets
Jun 21 06:54:52 dist-test-slave-gmf2 krb5kdc[1889](info): commencing operation
krb5kdc: starting...
W20260621 06:54:53.766353 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:54:53 dist-test-slave-gmf2 krb5kdc[1889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:54:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:54:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:54:53.977217 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:39511
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:39511,127.20.120.253:34313,127.20.120.252:36259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:54:54.294345 1902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:54.294665 1902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:54.294835 1902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:54.301573 1902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:54:54.301695 1902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:54.301798 1902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:54.301887 1902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:54:54.302000 1902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:54:54.311569 1902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:39511,127.20.120.253:34313,127.20.120.252:36259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:39511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:54.312870 1902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:54.314185 1902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:54:54.325968 1908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:54.326170 1910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:54.326579 1907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:54.326929 1902 server_base.cc:1034] running on GCE node
I20260621 06:54:54.327651 1902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:54.328909 1902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:54.330089 1902 hybrid_clock.cc:648] HybridClock initialized: now 1782024894330051 us; error 39 us; skew 500 ppm
I20260621 06:54:54.330442 1902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:54:54 dist-test-slave-gmf2 krb5kdc[1889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024894, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:54.333403 1902 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:54:54.334806 1902 webserver.cc:460] Webserver started at http://127.20.120.254:37189/ using document root <none> and password file <none>
I20260621 06:54:54.335446 1902 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:54.335541 1902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:54:54.335772 1902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:54:54.338634 1902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "1100de3d1f9b46a4a688935cefe24731"
format_stamp: "Formatted at 2026-06-21 06:54:54 on dist-test-slave-gmf2"
server_key: "704e6934c15708b42421921435a1f267"
server_key_iv: "885694334163e28e7537dc44edfa74f2"
server_key_version: "encryptionkey@0"
I20260621 06:54:54.339354 1902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "1100de3d1f9b46a4a688935cefe24731"
format_stamp: "Formatted at 2026-06-21 06:54:54 on dist-test-slave-gmf2"
server_key: "704e6934c15708b42421921435a1f267"
server_key_iv: "885694334163e28e7537dc44edfa74f2"
server_key_version: "encryptionkey@0"
I20260621 06:54:54.344141 1902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260621 06:54:54.347595 1917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:54.348799 1902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 06:54:54.348977 1902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "1100de3d1f9b46a4a688935cefe24731"
format_stamp: "Formatted at 2026-06-21 06:54:54 on dist-test-slave-gmf2"
server_key: "704e6934c15708b42421921435a1f267"
server_key_iv: "885694334163e28e7537dc44edfa74f2"
server_key_version: "encryptionkey@0"
I20260621 06:54:54.349153 1902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:54.360214 1902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:54.363865 1902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:54.364110 1902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:54.381616 1902 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:39511
I20260621 06:54:54.381649 1968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:39511 every 8 connection(s)
I20260621 06:54:54.382853 1902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:54:54.386233 1969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:54:54.386854 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 1902
I20260621 06:54:54.387075 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:54:54.387879 20963 external_mini_cluster.cc:1428] Setting key 5a64431eeb7d229e0e0bb83e1f8bd84d
I20260621 06:54:54.395975 1969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.120.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.253@KRBTEST.COM" created.
Jun 21 06:54:54 dist-test-slave-gmf2 krb5kdc[1889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024894, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:54.420451 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.398941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.20.120.254:39511)
0621 06:54:54.400514 (+ 1573us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:54.400526 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:54.400573 (+ 47us) server_negotiation.cc:407] Connection header received
0621 06:54:54.402023 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:54.402029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:54.402234 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:54.402434 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:54.403678 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:54.404822 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:54.406274 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:54.406767 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:54.406953 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:54.412844 (+ 5891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:54.412853 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:54.412865 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:54.412896 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:54.415736 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:54.416705 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:54.416711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:54.416713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:54.416856 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:54.417311 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:54.417314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:54.417316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:54.417833 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:54.418062 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:54.418595 (+ 533us) server_negotiation.cc:299] Negotiation successful
0621 06:54:54.418895 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":1383,"spinlock_wait_cycles":15744,"thread_start_us":1201,"threads_started":1}
I20260621 06:54:54.420475 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.398929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:39511 (local address 127.0.0.1:44944)
0621 06:54:54.399737 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.399788 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:54.401659 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:54.402452 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:54.402458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:54.403156 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:54.403479 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:54.403494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:54.404978 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:54.405007 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:54.406137 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:54.406149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:54.406990 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:54.406996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:54.407245 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:54.407907 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:54.407928 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:54.412608 (+ 4680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:54.415896 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:54.415911 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:54.415927 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:54.416545 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:54.416967 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:54.416974 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:54.416988 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:54.417207 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:54.417999 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:54.418007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:54.418219 (+ 212us) client_negotiation.cc:769] Sending connection context
0621 06:54:54.418454 (+ 235us) client_negotiation.cc:240] Negotiation successful
0621 06:54:54.419045 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":181,"thread_start_us":367,"threads_started":1}
I20260621 06:54:54.423844 1969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } has no permanent_uuid. Determining permanent_uuid...
I20260621 06:54:54.424937 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.424550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45036)
0621 06:54:54.424709 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.424808 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 06:54:54.425191 1919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
I20260621 06:54:54.426738 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.426403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45046)
0621 06:54:54.426565 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.426637 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 06:54:54.426966 1969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:34313: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.253.keytab.
Entry for principal kudu/127.20.120.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.253.keytab.
I20260621 06:54:54.437017 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.253:34313
--webserver_interface=127.20.120.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.253.keytab
--principal=kudu/127.20.120.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:39511,127.20.120.253:34313,127.20.120.252:36259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:54:54.476253 1969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } attempt: 1
I20260621 06:54:54.477429 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.476986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45062)
0621 06:54:54.477180 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.477302 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260621 06:54:54.478528 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.478215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45072)
0621 06:54:54.478349 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.478415 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 06:54:54.478761 1969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:34313: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
I20260621 06:54:54.547142 1969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } attempt: 2
I20260621 06:54:54.548322 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.547873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45074)
0621 06:54:54.548098 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.548186 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 06:54:54.549300 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.548975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45084)
0621 06:54:54.549114 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.549190 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 06:54:54.549536 1969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:34313: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
I20260621 06:54:54.640797 1969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } attempt: 3
I20260621 06:54:54.642040 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.641560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45094)
0621 06:54:54.641748 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.641849 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260621 06:54:54.643132 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.642751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45108)
0621 06:54:54.642884 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.642959 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 06:54:54.643419 1969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:34313: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
W20260621 06:54:54.752147 1976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:54.752461 1976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:54.752666 1976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:54.759558 1976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:54:54.759655 1976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:54.759699 1976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:54.759773 1976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:54:54.759840 1976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:54:54.769250 1976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:39511,127.20.120.253:34313,127.20.120.252:36259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.253.keytab
--principal=kudu/127.20.120.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.253:34313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.120.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:54.770521 1976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:54.771811 1976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:54:54.782953 1984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:54.783010 1981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:54.783035 1982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:54.784108 1976 server_base.cc:1034] running on GCE node
I20260621 06:54:54.784873 1976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:54.786304 1976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:54.786626 1969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } attempt: 4
I20260621 06:54:54.787554 1976 hybrid_clock.cc:648] HybridClock initialized: now 1782024894787516 us; error 73 us; skew 500 ppm
I20260621 06:54:54.787736 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.787200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45112)
0621 06:54:54.787481 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.787561 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260621 06:54:54.787962 1976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:54:54.788717 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.788329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45126)
0621 06:54:54.788484 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.788573 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 06:54:54.788924 1969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:34313: Network error: Client connection negotiation failed: client connection to 127.20.120.253:34313: connect: Connection refused (error 111)
Jun 21 06:54:54 dist-test-slave-gmf2 krb5kdc[1889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024894, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:54.791042 1976 init.cc:377] Logged in from keytab as kudu/127.20.120.253@KRBTEST.COM (short username kudu)
I20260621 06:54:54.792532 1976 webserver.cc:460] Webserver started at http://127.20.120.253:45479/ using document root <none> and password file <none>
I20260621 06:54:54.793229 1976 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:54.793375 1976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:54:54.793676 1976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:54:54.796617 1976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/data/instance:
uuid: "06d64c55f4fb42f2a54b28eef5f3c83f"
format_stamp: "Formatted at 2026-06-21 06:54:54 on dist-test-slave-gmf2"
server_key: "86d9e9d0987a28811b7a11f668727a4f"
server_key_iv: "9a0a81988e560ce8dcb0f3d356d1e48d"
server_key_version: "encryptionkey@0"
I20260621 06:54:54.797444 1976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/wal/instance:
uuid: "06d64c55f4fb42f2a54b28eef5f3c83f"
format_stamp: "Formatted at 2026-06-21 06:54:54 on dist-test-slave-gmf2"
server_key: "86d9e9d0987a28811b7a11f668727a4f"
server_key_iv: "9a0a81988e560ce8dcb0f3d356d1e48d"
server_key_version: "encryptionkey@0"
I20260621 06:54:54.802345 1976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 06:54:54.805667 1991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:54.806948 1976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:54:54.807144 1976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/wal
uuid: "06d64c55f4fb42f2a54b28eef5f3c83f"
format_stamp: "Formatted at 2026-06-21 06:54:54 on dist-test-slave-gmf2"
server_key: "86d9e9d0987a28811b7a11f668727a4f"
server_key_iv: "9a0a81988e560ce8dcb0f3d356d1e48d"
server_key_version: "encryptionkey@0"
I20260621 06:54:54.807343 1976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:54.829237 1976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:54.833225 1976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:54.833516 1976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:54.852703 2042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.253:34313 every 8 connection(s)
I20260621 06:54:54.852715 1976 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.253:34313
I20260621 06:54:54.854298 1976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
I20260621 06:54:54.856742 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 1976
I20260621 06:54:54.856964 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-1/wal/instance
I20260621 06:54:54.857426 20963 external_mini_cluster.cc:1428] Setting key acf3c3fab25002ab31503bdc42585065
I20260621 06:54:54.858527 2043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:54:54.865406 2043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.120.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.252@KRBTEST.COM" created.
Jun 21 06:54:54 dist-test-slave-gmf2 krb5kdc[1889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024894, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.253@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:54.888413 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.868311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.20.120.254:39511)
0621 06:54:54.868599 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:54.868608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:54.869982 (+ 1374us) server_negotiation.cc:407] Connection header received
0621 06:54:54.872007 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:54.872015 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:54.872115 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:54.872246 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:54.873858 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:54.875033 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:54.876691 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:54.877242 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:54.877344 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:54.881327 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:54.881336 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:54.881344 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:54.881393 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:54.885144 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:54.886019 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:54.886027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:54.886034 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:54.886187 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:54.886650 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:54.886657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:54.886660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:54.886957 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:54.887111 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:54.887865 (+ 754us) server_negotiation.cc:299] Negotiation successful
0621 06:54:54.888089 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 06:54:54.889479 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.868869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:39511 (local address 127.0.0.1:44956)
0621 06:54:54.869747 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.869815 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:54.871697 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:54.872476 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:54.872484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:54.873077 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:54.873661 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:54.873680 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:54.875219 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:54.875223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:54.876553 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:54.876566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:54.877460 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:54.877466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:54.877659 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:54.878396 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:54.878422 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:54.881048 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:54.885326 (+ 4278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:54.885333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:54.885347 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:54.885814 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:54.886325 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:54.886328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:54.886331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:54.886516 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:54.887099 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:54.887107 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:54.887478 (+ 371us) client_negotiation.cc:769] Sending connection context
0621 06:54:54.887780 (+ 302us) client_negotiation.cc:240] Negotiation successful
0621 06:54:54.888295 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":384,"threads_started":1}
I20260621 06:54:54.891319 2043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:54:54 dist-test-slave-gmf2 krb5kdc[1889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024894, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.253@KRBTEST.COM for kudu/127.20.120.253@KRBTEST.COM
I20260621 06:54:54.907033 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.892161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45134)
0621 06:54:54.892343 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.892370 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:54.892490 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:54.893351 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:54.893354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:54.893699 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:54.893834 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:54.893841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:54.895916 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:54.895920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:54.896753 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:54.896763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:54.898083 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:54.898087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:54.898274 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:54.898851 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:54.898866 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:54.900739 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:54.904274 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:54.904278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:54.904280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:54.904742 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:54.905179 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:54.905187 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:54.905192 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:54.905366 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:54.906176 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:54.906180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:54.906315 (+ 135us) client_negotiation.cc:769] Sending connection context
0621 06:54:54.906441 (+ 126us) client_negotiation.cc:240] Negotiation successful
0621 06:54:54.906652 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 06:54:54.907231 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.892179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.20.120.253:34313)
0621 06:54:54.892833 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:54.892843 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:54.892875 (+ 32us) server_negotiation.cc:407] Connection header received
0621 06:54:54.892931 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:54.892941 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:54.893199 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:54.893350 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:54.894043 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:54.895784 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:54.896890 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:54.897940 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:54.898094 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:54.900942 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:54.900952 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:54.900966 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:54.901000 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:54.904139 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:54.904854 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:54.904861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:54.904863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:54.905021 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:54.905470 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:54.905474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:54.905476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:54.906026 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:54.906250 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:54.906586 (+ 336us) server_negotiation.cc:299] Negotiation successful
0621 06:54:54.906855 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":264,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.252.keytab.
Entry for principal kudu/127.20.120.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.252.keytab.
I20260621 06:54:54.909994 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.252:36259
--webserver_interface=127.20.120.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.252.keytab
--principal=kudu/127.20.120.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:39511,127.20.120.253:34313,127.20.120.252:36259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:54:54.909957 2043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } has no permanent_uuid. Determining permanent_uuid...
I20260621 06:54:54.911298 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.910842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53526)
0621 06:54:54.911032 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.911146 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260621 06:54:54.911715 1995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
I20260621 06:54:54.913403 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.913012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53538)
0621 06:54:54.913193 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.913269 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260621 06:54:54.913666 2043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36259: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
I20260621 06:54:54.963006 2043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } attempt: 1
I20260621 06:54:54.964231 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.963738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53548)
0621 06:54:54.963955 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.964067 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260621 06:54:54.965293 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:54.964950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53554)
0621 06:54:54.965129 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:54.965192 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260621 06:54:54.965548 2043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36259: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
I20260621 06:54:55.033802 2043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } attempt: 2
I20260621 06:54:55.035024 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.034538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53556)
0621 06:54:55.034761 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.034861 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260621 06:54:55.036067 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.035772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53562)
0621 06:54:55.035896 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.035962 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 06:54:55.036597 2043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36259: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
I20260621 06:54:55.088207 1969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } attempt: 5
Jun 21 06:54:55 dist-test-slave-gmf2 krb5kdc[1889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024894, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for kudu/127.20.120.253@KRBTEST.COM
I20260621 06:54:55.103606 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.089175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45150 (local address 127.20.120.253:34313)
0621 06:54:55.089338 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:55.089342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:55.089363 (+ 21us) server_negotiation.cc:407] Connection header received
0621 06:54:55.089558 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:55.089562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:55.089643 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:55.089732 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:55.091036 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.091710 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.093673 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.094203 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.094306 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.098171 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:55.098176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:55.098179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:55.098214 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:55.100909 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.101624 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.101630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.101633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.101761 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.102185 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.102191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.102194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.102484 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:55.102644 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:55.103049 (+ 405us) server_negotiation.cc:299] Negotiation successful
0621 06:54:55.103277 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 06:54:55.103619 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.089016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45150)
0621 06:54:55.089254 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.089277 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:55.089418 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:55.089832 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:55.089839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:55.090548 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:55.090822 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:55.090834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.091854 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.091859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:55.093491 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:55.093511 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.094375 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.094383 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:55.094627 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.095394 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:55.095418 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:55.097898 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:55.101103 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:55.101109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:55.101113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:55.101494 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:55.101878 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:55.101883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:55.101887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:55.102073 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:55.102612 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:55.102617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:55.102782 (+ 165us) client_negotiation.cc:769] Sending connection context
0621 06:54:55.102925 (+ 143us) client_negotiation.cc:240] Negotiation successful
0621 06:54:55.103256 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260621 06:54:55.104815 1969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } has no permanent_uuid. Determining permanent_uuid...
I20260621 06:54:55.106148 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.105693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53572)
0621 06:54:55.105870 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.106013 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260621 06:54:55.107270 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.106930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53574)
0621 06:54:55.107093 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.107162 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260621 06:54:55.107513 1969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36259: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
I20260621 06:54:55.127851 2043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } attempt: 3
I20260621 06:54:55.129055 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.128553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53578)
0621 06:54:55.128760 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.128867 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260621 06:54:55.130223 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.129775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53590)
0621 06:54:55.130027 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.130111 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260621 06:54:55.130457 2043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36259: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
I20260621 06:54:55.158743 1969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } attempt: 1
I20260621 06:54:55.159971 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.159489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53602)
0621 06:54:55.159703 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.159811 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260621 06:54:55.161012 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.160630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53606)
0621 06:54:55.160810 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.160888 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 06:54:55.161219 1969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36259: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
I20260621 06:54:55.229444 1969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } attempt: 2
I20260621 06:54:55.230669 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.230145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53608)
0621 06:54:55.230401 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.230505 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260621 06:54:55.231704 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.231362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53616)
0621 06:54:55.231511 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.231595 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 06:54:55.231935 1969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36259: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
W20260621 06:54:55.244324 2050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:55.244688 2050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:55.244887 2050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:55.252853 2050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:54:55.252995 2050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:55.253129 2050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:55.253235 2050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:54:55.253337 2050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:54:55.263665 2050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:39511,127.20.120.253:34313,127.20.120.252:36259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.252.keytab
--principal=kudu/127.20.120.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.252:36259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.120.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:55.265064 2050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:55.266577 2050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 06:54:55.273697 2043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } attempt: 4
I20260621 06:54:55.274881 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.274391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53630)
0621 06:54:55.274619 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.274713 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260621 06:54:55.275853 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.275552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53646)
0621 06:54:55.275682 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.275749 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 06:54:55.276151 2043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36259: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
W20260621 06:54:55.279212 2056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:55.279177 2058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:55.279536 2055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:55.280908 2050 server_base.cc:1034] running on GCE node
I20260621 06:54:55.281664 2050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:55.283164 2050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:55.284437 2050 hybrid_clock.cc:648] HybridClock initialized: now 1782024895284388 us; error 54 us; skew 500 ppm
I20260621 06:54:55.284824 2050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:54:55 dist-test-slave-gmf2 krb5kdc[1889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024895, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:55.287869 2050 init.cc:377] Logged in from keytab as kudu/127.20.120.252@KRBTEST.COM (short username kudu)
I20260621 06:54:55.289240 2050 webserver.cc:460] Webserver started at http://127.20.120.252:45695/ using document root <none> and password file <none>
I20260621 06:54:55.289896 2050 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:55.290086 2050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:54:55.290378 2050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:54:55.293359 2050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/data/instance:
uuid: "266e1bdabd1b441196cc186177f2db75"
format_stamp: "Formatted at 2026-06-21 06:54:55 on dist-test-slave-gmf2"
server_key: "82b700c2b57b216d66caacaa09ecdb26"
server_key_iv: "55d4df2797f3eb3c97bf32c75936ac6f"
server_key_version: "encryptionkey@0"
I20260621 06:54:55.294214 2050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/wal/instance:
uuid: "266e1bdabd1b441196cc186177f2db75"
format_stamp: "Formatted at 2026-06-21 06:54:55 on dist-test-slave-gmf2"
server_key: "82b700c2b57b216d66caacaa09ecdb26"
server_key_iv: "55d4df2797f3eb3c97bf32c75936ac6f"
server_key_version: "encryptionkey@0"
I20260621 06:54:55.299885 2050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:54:55.304206 2065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:55.305960 2050 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260621 06:54:55.306144 2050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/wal
uuid: "266e1bdabd1b441196cc186177f2db75"
format_stamp: "Formatted at 2026-06-21 06:54:55 on dist-test-slave-gmf2"
server_key: "82b700c2b57b216d66caacaa09ecdb26"
server_key_iv: "55d4df2797f3eb3c97bf32c75936ac6f"
server_key_version: "encryptionkey@0"
I20260621 06:54:55.306335 2050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:55.338130 1969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } attempt: 3
I20260621 06:54:55.339335 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.338821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53650)
0621 06:54:55.339039 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.339145 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 06:54:55.340401 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.339998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53666)
0621 06:54:55.340194 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.340279 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260621 06:54:55.340641 1969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36259: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36259: connect: Connection refused (error 111)
I20260621 06:54:55.345132 2050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:55.349040 2050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:55.349321 2050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:55.367506 2050 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.252:36259
I20260621 06:54:55.367545 2116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.252:36259 every 8 connection(s)
I20260621 06:54:55.368811 2050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
I20260621 06:54:55.372393 2117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:54:55.373811 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 2050
I20260621 06:54:55.374032 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782024700637160-20963-0/minicluster-data/master-2/wal/instance
I20260621 06:54:55.374598 20963 external_mini_cluster.cc:1428] Setting key a89d2ae89f510b474ce0868023c6f10c
I20260621 06:54:55.381731 2117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:54:55 dist-test-slave-gmf2 krb5kdc[1889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.252@KRBTEST.COM
Jun 21 06:54:55 dist-test-slave-gmf2 krb5kdc[1889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024895, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.252@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:55.399031 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.376391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.20.120.252:36259)
0621 06:54:55.377174 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:55.377184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:55.377222 (+ 38us) server_negotiation.cc:407] Connection header received
0621 06:54:55.378849 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:55.378868 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:55.379254 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:55.379551 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:55.381220 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.383270 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.385108 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.386125 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.386296 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.391041 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:55.391058 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:55.391068 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:55.391110 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:55.394674 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.395511 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.395521 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.395531 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.395724 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.396159 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.396162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.396164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.396735 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:55.396923 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:55.397414 (+ 491us) server_negotiation.cc:299] Negotiation successful
0621 06:54:55.397683 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":366,"threads_started":1}
I20260621 06:54:55.401510 2121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.384129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:39511 (local address 127.0.0.1:44964)
0621 06:54:55.384859 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.384907 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:55.385127 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:55.385510 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:55.385517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:55.386143 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:55.386527 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:55.386540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.387894 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.387901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:55.389425 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:55.389436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.390366 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.390375 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:55.390544 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.391669 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:55.391696 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:55.394478 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:55.398144 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:55.398153 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:55.398159 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:55.398674 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:55.399039 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:55.399046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:55.399052 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:55.399980 (+ 928us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:55.400569 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:55.400574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:55.400713 (+ 139us) client_negotiation.cc:769] Sending connection context
0621 06:54:55.400856 (+ 143us) client_negotiation.cc:240] Negotiation successful
0621 06:54:55.401186 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":379,"threads_started":1}
I20260621 06:54:55.401893 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.384184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.20.120.254:39511)
0621 06:54:55.384351 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:55.384356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:55.385003 (+ 647us) server_negotiation.cc:407] Connection header received
0621 06:54:55.385289 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:55.385293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:55.385353 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:55.385463 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:55.386682 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.387717 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.389563 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.390144 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.390286 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.394665 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:55.394670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:55.394674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:55.394704 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:55.398033 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.398783 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.398787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.398789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.398909 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.400112 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.400117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.400119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.400457 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:55.400603 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:55.400993 (+ 390us) server_negotiation.cc:299] Negotiation successful
0621 06:54:55.401231 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 06:54:55.403110 2117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:54:55 dist-test-slave-gmf2 krb5kdc[1889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024895, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.252@KRBTEST.COM for kudu/127.20.120.253@KRBTEST.COM
Jun 21 06:54:55 dist-test-slave-gmf2 krb5kdc[1889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:55.416456 2121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.403805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:34313 (local address 127.0.0.1:45164)
0621 06:54:55.403947 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.403965 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:55.404050 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:55.404380 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:55.404383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:55.404680 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:55.404789 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:55.404795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.405789 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.405794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:55.406923 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:55.406947 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.407634 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.407639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:55.407850 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.408350 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:55.408364 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:55.410410 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:55.414162 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:55.414166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:55.414168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:55.414529 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:55.414926 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:55.414931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:55.414934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:55.415098 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:55.415689 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:55.415694 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:55.415857 (+ 163us) client_negotiation.cc:769] Sending connection context
0621 06:54:55.416013 (+ 156us) client_negotiation.cc:240] Negotiation successful
0621 06:54:55.416196 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 06:54:55.416883 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.403824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.20.120.253:34313)
0621 06:54:55.404033 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:55.404039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:55.404057 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:54:55.404180 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:55.404186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:55.404261 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:55.404362 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:55.404928 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.405634 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.407085 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.407469 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.407625 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.410616 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:55.410622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:55.410627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:55.410665 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:55.414038 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.414644 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.414651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.414658 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.414818 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.415220 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.415223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.415225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.415573 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:55.415713 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:55.416124 (+ 411us) server_negotiation.cc:299] Negotiation successful
0621 06:54:55.416324 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 06:54:55.417529 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.402014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.20.120.254:39511)
0621 06:54:55.402444 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:55.402450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:55.402467 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:54:55.402516 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:55.402520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:55.402590 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:55.402707 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:55.403938 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.404841 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.406323 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.406705 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.406807 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.411350 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:55.411356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:55.411360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:55.411394 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:55.414616 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.415422 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.415427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.415430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.415567 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.415968 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.415971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.415973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.416313 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:55.416581 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:55.416922 (+ 341us) server_negotiation.cc:299] Negotiation successful
0621 06:54:55.417175 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":413,"threads_started":1}
I20260621 06:54:55.418016 2117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:54:55 dist-test-slave-gmf2 krb5kdc[1889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024895, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.252@KRBTEST.COM for kudu/127.20.120.252@KRBTEST.COM
Jun 21 06:54:55 dist-test-slave-gmf2 krb5kdc[1889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.253@KRBTEST.COM
I20260621 06:54:55.431232 2121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.419017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:53694)
0621 06:54:55.419252 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.419293 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:55.419397 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:55.419696 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:55.419700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:55.420107 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:55.420245 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:55.420253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.421145 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.421153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:55.422575 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:55.422586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.423254 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.423260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:55.423459 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.424017 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:55.424039 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:55.426161 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:55.429087 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:55.429093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:55.429097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:55.429416 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:55.429801 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:55.429804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:55.429806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:55.429965 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:55.430545 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:55.430548 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:55.430652 (+ 104us) client_negotiation.cc:769] Sending connection context
0621 06:54:55.430759 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 06:54:55.430941 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260621 06:54:55.431514 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.418962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.20.120.252:36259)
0621 06:54:55.419114 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:55.419122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:55.419353 (+ 231us) server_negotiation.cc:407] Connection header received
0621 06:54:55.419525 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:55.419530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:55.419588 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:55.419683 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:55.420378 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.421025 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.422706 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.423116 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.423237 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.426335 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:55.426339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:55.426342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:55.426367 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:55.428951 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.429546 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.429550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.429552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.429694 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.430070 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.430074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.430076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.430420 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:55.430577 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:55.430905 (+ 328us) server_negotiation.cc:299] Negotiation successful
0621 06:54:55.431130 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 06:54:55.433846 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.419404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.20.120.253:34313)
0621 06:54:55.419567 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:55.419573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:55.419619 (+ 46us) server_negotiation.cc:407] Connection header received
0621 06:54:55.419795 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:55.419800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:55.419865 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:55.420013 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:55.421278 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.422400 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.423587 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.424227 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.424357 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.428230 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:55.428235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:55.428240 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:55.428281 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:55.431141 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.431925 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.431932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.431943 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.432090 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.432455 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.432478 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.432482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.432824 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:55.433116 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:55.433308 (+ 192us) server_negotiation.cc:299] Negotiation successful
0621 06:54:55.433523 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 06:54:55.437400 2117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75: Bootstrap starting.
I20260621 06:54:55.439736 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:54:55.441824 2117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75: Neither blocks nor log segments found. Creating new log.
I20260621 06:54:55.443228 2117 log.cc:824] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75: Log is configured to *not* fsync() on all Append() calls
I20260621 06:54:55.446943 2117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75: No bootstrap required, opened a new log
I20260621 06:54:55.458420 2117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } }
I20260621 06:54:55.459138 2117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:54:55.459317 2117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266e1bdabd1b441196cc186177f2db75, State: Initialized, Role: FOLLOWER
I20260621 06:54:55.460073 2117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } }
I20260621 06:54:55.467031 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.446065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.20.120.253:34313)
0621 06:54:55.446214 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:55.446219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:55.446805 (+ 586us) server_negotiation.cc:407] Connection header received
0621 06:54:55.448059 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:55.448065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:55.448156 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:55.448236 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:55.450584 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.451699 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.454470 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.455108 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.455203 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.458985 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:55.458991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:55.458995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:55.459031 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:55.462913 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.464049 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.464057 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.464063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.464247 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.464730 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.464737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.464743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.465255 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:55.465539 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:55.466308 (+ 769us) server_negotiation.cc:299] Negotiation successful
0621 06:54:55.466615 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 06:54:55.467222 2131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } } }
I20260621 06:54:55.468367 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.446012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.20.120.254:39511)
0621 06:54:55.446188 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:55.446195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:55.448086 (+ 1891us) server_negotiation.cc:407] Connection header received
0621 06:54:55.448411 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:55.448419 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:55.448533 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:55.448616 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:55.451374 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.452809 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.455682 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.456546 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.456659 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.461524 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:55.461533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:55.461537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:55.461582 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:55.465117 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.466021 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.466027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.466030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.466192 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.466681 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.466686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.466689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.467012 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:55.467234 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:55.467771 (+ 537us) server_negotiation.cc:299] Negotiation successful
0621 06:54:55.468015 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 06:54:55.469784 2131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:54:55.469645 2117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:54:55.477445 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.447196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.20.120.252:36259)
0621 06:54:55.447366 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:55.447374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:55.461193 (+ 13819us) server_negotiation.cc:407] Connection header received
0621 06:54:55.461615 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:55.461622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:55.461702 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:55.461769 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:55.463643 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.464783 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.466833 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.467573 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.467660 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.471076 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:55.471084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:55.471092 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:55.471139 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:55.474786 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.475399 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.475409 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.475411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.475567 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.475959 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.475962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.475964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.476241 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:55.476550 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:55.476824 (+ 274us) server_negotiation.cc:299] Negotiation successful
0621 06:54:55.477072 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260621 06:54:55.481812 2142 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 06:54:55.482018 2142 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 06:54:55.479775 2050 master_runner.cc:186] Error getting master registration for 127.20.120.254:39511: OK, instance_id { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" instance_seqno: 1782024894376676 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 06:54:55.490653 2050 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 06:54:55.517966 1969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } attempt: 4
Jun 21 06:54:55 dist-test-slave-gmf2 krb5kdc[1889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024894, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for kudu/127.20.120.252@KRBTEST.COM
I20260621 06:54:55.533095 1972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.518639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:57196)
0621 06:54:55.518837 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.518859 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:55.518964 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:55.519298 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:55.519303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:55.519734 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:55.519908 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:55.519917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.521017 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.521024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:55.521997 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:55.522008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.523469 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.523475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:55.523694 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.524493 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:55.524512 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:55.526744 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:55.530583 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:55.530587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:55.530589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:55.530915 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:55.531277 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:55.531280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:55.531282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:55.531407 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:55.532056 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:55.532061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:55.532231 (+ 170us) client_negotiation.cc:769] Sending connection context
0621 06:54:55.532394 (+ 163us) client_negotiation.cc:240] Negotiation successful
0621 06:54:55.532671 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260621 06:54:55.534179 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.518702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.20.120.252:36259)
0621 06:54:55.518871 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:55.518879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:55.518919 (+ 40us) server_negotiation.cc:407] Connection header received
0621 06:54:55.519101 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:55.519105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:55.519172 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:55.519275 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:55.520062 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.520885 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.522137 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.523294 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.524074 (+ 780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.526923 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:55.526930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:55.526936 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:55.526972 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:55.530462 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.531006 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.531011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.531015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.531171 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.531516 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.531526 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.531530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.531886 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:55.532054 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:55.533611 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0621 06:54:55.533816 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 06:54:55.539875 1969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731: Bootstrap starting.
I20260621 06:54:55.543998 1969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731: Neither blocks nor log segments found. Creating new log.
I20260621 06:54:55.545245 1969 log.cc:824] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731: Log is configured to *not* fsync() on all Append() calls
I20260621 06:54:55.548576 1969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731: No bootstrap required, opened a new log
I20260621 06:54:55.559269 1969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } }
I20260621 06:54:55.559643 1969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:54:55.559756 1969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1100de3d1f9b46a4a688935cefe24731, State: Initialized, Role: FOLLOWER
I20260621 06:54:55.560400 1969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } }
I20260621 06:54:55.561223 2145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } } }
I20260621 06:54:55.561553 2145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:54:55.562573 1969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:54:55.575384 2043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } attempt: 5
W20260621 06:54:55.575910 2156 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 06:54:55.576088 2156 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 06:54:55.571022 1902 master_runner.cc:186] Error getting master registration for 127.20.120.253:34313: OK, instance_id { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" instance_seqno: 1782024894847078 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 06:54:55.582286 1902 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 21 06:54:55 dist-test-slave-gmf2 krb5kdc[1889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024894, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.253@KRBTEST.COM for kudu/127.20.120.252@KRBTEST.COM
I20260621 06:54:55.591050 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.575970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36259 (local address 127.0.0.1:57198)
0621 06:54:55.576287 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0621 06:54:55.576305 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:54:55.576450 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:54:55.576970 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:54:55.576978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:54:55.577473 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:54:55.577674 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:55.577687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.579158 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.579165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:55.580455 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:54:55.580470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.581531 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.581536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:54:55.581796 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.582485 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:54:55.582502 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:54:55.584504 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:54:55.588299 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:55.588303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:55.588307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:55.588742 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:55.589160 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:54:55.589164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:54:55.589168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:54:55.589341 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:54:55.590037 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:54:55.590044 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:54:55.590187 (+ 143us) client_negotiation.cc:769] Sending connection context
0621 06:54:55.590380 (+ 193us) client_negotiation.cc:240] Negotiation successful
0621 06:54:55.590632 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260621 06:54:55.591065 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:55.576259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.20.120.252:36259)
0621 06:54:55.576421 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:55.576426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:55.576445 (+ 19us) server_negotiation.cc:407] Connection header received
0621 06:54:55.576608 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:55.576614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:55.576689 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:55.576773 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:55.577857 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.578961 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.580603 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:55.581244 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:55.581359 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:55.584696 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:55.584701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:55.584704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:55.584735 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:55.588153 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.588897 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.588901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.588904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.589037 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:55.589464 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:55.589468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:55.589470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:55.589787 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:55.589957 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:55.590462 (+ 505us) server_negotiation.cc:299] Negotiation successful
0621 06:54:55.590688 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 06:54:55.595530 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f: Bootstrap starting.
I20260621 06:54:55.598413 2043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f: Neither blocks nor log segments found. Creating new log.
I20260621 06:54:55.599390 2043 log.cc:824] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f: Log is configured to *not* fsync() on all Append() calls
W20260621 06:54:55.601380 2071 tablet.cc:2369] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:54:55.602125 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f: No bootstrap required, opened a new log
I20260621 06:54:55.610121 2043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } }
I20260621 06:54:55.610532 2043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:54:55.610669 2043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d64c55f4fb42f2a54b28eef5f3c83f, State: Initialized, Role: FOLLOWER
I20260621 06:54:55.611224 2043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } }
I20260621 06:54:55.612197 2159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } } }
I20260621 06:54:55.612612 2159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:54:55.613310 2043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [sys.catalog]: configured and running, proceeding with master startup.
W20260621 06:54:55.616715 1923 tablet.cc:2369] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:54:55.629894 1976 master_runner.cc:263] No leader master found from master 06d64c55f4fb42f2a54b28eef5f3c83f
I20260621 06:54:55.630100 1976 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260621 06:54:55.632690 2170 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 06:54:55.632833 2170 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 06:54:55.698995 2145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:54:55.699296 2145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } }
I20260621 06:54:55.701010 2145 leader_election.cc:288] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06d64c55f4fb42f2a54b28eef5f3c83f (127.20.120.253:34313), 266e1bdabd1b441196cc186177f2db75 (127.20.120.252:36259)
I20260621 06:54:55.702170 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1100de3d1f9b46a4a688935cefe24731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "266e1bdabd1b441196cc186177f2db75" is_pre_election: true
I20260621 06:54:55.702806 2092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1100de3d1f9b46a4a688935cefe24731 in term 0.
I20260621 06:54:55.703563 1921 leader_election.cc:302] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1100de3d1f9b46a4a688935cefe24731, 266e1bdabd1b441196cc186177f2db75; no voters:
I20260621 06:54:55.704720 2018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1100de3d1f9b46a4a688935cefe24731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" is_pre_election: true
I20260621 06:54:55.705302 2018 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1100de3d1f9b46a4a688935cefe24731 in term 0.
I20260621 06:54:55.706194 2145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:54:55.706341 2145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:54:55.706444 2145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:54:55.710106 2145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } }
I20260621 06:54:55.715263 2018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1100de3d1f9b46a4a688935cefe24731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f"
I20260621 06:54:55.715493 2018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:54:55.718988 2018 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100de3d1f9b46a4a688935cefe24731 in term 1.
I20260621 06:54:55.719465 1919 leader_election.cc:302] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d64c55f4fb42f2a54b28eef5f3c83f, 1100de3d1f9b46a4a688935cefe24731; no voters:
I20260621 06:54:55.722715 2092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1100de3d1f9b46a4a688935cefe24731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "266e1bdabd1b441196cc186177f2db75"
I20260621 06:54:55.722951 2092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:54:55.726374 2092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1100de3d1f9b46a4a688935cefe24731 in term 1.
I20260621 06:54:55.727006 2145 leader_election.cc:288] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [CANDIDATE]: Term 1 election: Requested vote from peers 06d64c55f4fb42f2a54b28eef5f3c83f (127.20.120.253:34313), 266e1bdabd1b441196cc186177f2db75 (127.20.120.252:36259)
I20260621 06:54:55.727370 2145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:54:55.727747 2145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [term 1 LEADER]: Becoming Leader. State: Replica: 1100de3d1f9b46a4a688935cefe24731, State: Running, Role: LEADER
I20260621 06:54:55.728538 2145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } }
I20260621 06:54:55.747534 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1100de3d1f9b46a4a688935cefe24731. Latest consensus state: current_term: 1 leader_uuid: "1100de3d1f9b46a4a688935cefe24731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } } }
I20260621 06:54:55.748102 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [sys.catalog]: This master's current role is: LEADER
I20260621 06:54:55.755146 2174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:54:55.757433 2174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:54:55.777338 2092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [term 1 FOLLOWER]: Refusing update from remote peer 1100de3d1f9b46a4a688935cefe24731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:54:55.778393 2172 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:54:55.785485 2018 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [term 1 FOLLOWER]: Refusing update from remote peer 1100de3d1f9b46a4a688935cefe24731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:54:55.786307 2172 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:54:55.823810 2159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1100de3d1f9b46a4a688935cefe24731. Latest consensus state: current_term: 1 leader_uuid: "1100de3d1f9b46a4a688935cefe24731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } } }
I20260621 06:54:55.824151 2159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:54:55.828207 2175 mvcc.cc:205] Tried to move back new op lower bound from 7299173973072945152 to 7299173972915593216. Current Snapshot: MvccSnapshot[applied={T|T < 7299173973072945152}]
I20260621 06:54:55.833488 2177 mvcc.cc:205] Tried to move back new op lower bound from 7299173973072945152 to 7299173972915593216. Current Snapshot: MvccSnapshot[applied={T|T < 7299173973072945152}]
I20260621 06:54:55.838747 2145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1100de3d1f9b46a4a688935cefe24731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } } }
I20260621 06:54:55.839087 2145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [sys.catalog]: This master's current role is: LEADER
I20260621 06:54:55.845227 2174 catalog_manager.cc:1347] Generated new cluster ID: d704bc47e6f04c48823e19c09ce673a6
I20260621 06:54:55.845337 2174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:54:55.848748 2159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1100de3d1f9b46a4a688935cefe24731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } } }
I20260621 06:54:55.848989 2159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:54:55.876495 2131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1100de3d1f9b46a4a688935cefe24731. Latest consensus state: current_term: 1 leader_uuid: "1100de3d1f9b46a4a688935cefe24731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } } }
I20260621 06:54:55.876799 2131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:54:55.888481 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1100de3d1f9b46a4a688935cefe24731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } } }
I20260621 06:54:55.889744 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731 [sys.catalog]: This master's current role is: LEADER
I20260621 06:54:55.894863 2176 mvcc.cc:205] Tried to move back new op lower bound from 7299173973072945152 to 7299173972915593216. Current Snapshot: MvccSnapshot[applied={T|T < 7299173973072945152}]
I20260621 06:54:55.907326 2131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1100de3d1f9b46a4a688935cefe24731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1100de3d1f9b46a4a688935cefe24731" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39511 } } peers { permanent_uuid: "06d64c55f4fb42f2a54b28eef5f3c83f" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 34313 } } peers { permanent_uuid: "266e1bdabd1b441196cc186177f2db75" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36259 } } }
I20260621 06:54:55.907608 2131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:54:55.935595 2174 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:54:55.937593 2174 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:54:55.993458 2174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1100de3d1f9b46a4a688935cefe24731: Generated new TSK 0
I20260621 06:54:55.994753 2174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:54:56.483944 2142 catalog_manager.cc:1259] Loaded cluster ID: d704bc47e6f04c48823e19c09ce673a6
I20260621 06:54:56.484098 2142 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75: loading cluster ID for follower catalog manager: success
I20260621 06:54:56.486752 2142 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75: acquiring CA information for follower catalog manager: success
I20260621 06:54:56.488575 2142 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 266e1bdabd1b441196cc186177f2db75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 06:54:56.634851 2170 catalog_manager.cc:1259] Loaded cluster ID: d704bc47e6f04c48823e19c09ce673a6
I20260621 06:54:56.634948 2170 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f: loading cluster ID for follower catalog manager: success
I20260621 06:54:56.637308 2170 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f: acquiring CA information for follower catalog manager: success
I20260621 06:54:56.639097 2170 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 06d64c55f4fb42f2a54b28eef5f3c83f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 06:54:57.104162 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 1902
I20260621 06:54:57.105326 1964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:57.216138 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 1902
I20260621 06:54:57.237047 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 1976
I20260621 06:54:57.238142 2038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:57.343690 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 1976
I20260621 06:54:57.363758 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 2050
I20260621 06:54:57.364971 2112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:57.475045 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 2050
2026-06-21T06:54:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4806 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:54:57 dist-test-slave-gmf2 krb5kdc[2194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:54:57 dist-test-slave-gmf2 krb5kdc[2194](info): set up 2 sockets
Jun 21 06:54:57 dist-test-slave-gmf2 krb5kdc[2194](info): commencing operation
krb5kdc: starting...
W20260621 06:54:58.550457 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:54:58 dist-test-slave-gmf2 krb5kdc[2194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:54:58.760526 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:39383
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:39383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:54:59.083297 2207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:54:59.083603 2207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:54:59.083755 2207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:54:59.090736 2207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:54:59.090871 2207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:54:59.090992 2207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:54:59.091089 2207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:54:59.091173 2207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:54:59.100502 2207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:39383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:39383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:54:59.101783 2207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:54:59.103147 2207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:54:59.114923 2215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:59.115072 2212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:54:59.115041 2213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:54:59.116003 2207 server_base.cc:1034] running on GCE node
I20260621 06:54:59.116828 2207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:54:59.118427 2207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:54:59.119675 2207 hybrid_clock.cc:648] HybridClock initialized: now 1782024899119620 us; error 42 us; skew 500 ppm
I20260621 06:54:59.120055 2207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:54:59 dist-test-slave-gmf2 krb5kdc[2194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024899, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:54:59.126190 2207 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:54:59.127780 2207 webserver.cc:460] Webserver started at http://127.20.120.254:36181/ using document root <none> and password file <none>
I20260621 06:54:59.128578 2207 fs_manager.cc:359] Metadata directory not provided
I20260621 06:54:59.128727 2207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:54:59.129050 2207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:54:59.132120 2207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "0355f94517a841c1a02e375741c41fa7"
format_stamp: "Formatted at 2026-06-21 06:54:59 on dist-test-slave-gmf2"
server_key: "8f612a72571f3d503cf1415fa42afaff"
server_key_iv: "3198d433dbfb688998d3ec455ca4c12a"
server_key_version: "encryptionkey@0"
I20260621 06:54:59.132861 2207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "0355f94517a841c1a02e375741c41fa7"
format_stamp: "Formatted at 2026-06-21 06:54:59 on dist-test-slave-gmf2"
server_key: "8f612a72571f3d503cf1415fa42afaff"
server_key_iv: "3198d433dbfb688998d3ec455ca4c12a"
server_key_version: "encryptionkey@0"
I20260621 06:54:59.138197 2207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 06:54:59.141557 2222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:54:59.142937 2207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:54:59.143137 2207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "0355f94517a841c1a02e375741c41fa7"
format_stamp: "Formatted at 2026-06-21 06:54:59 on dist-test-slave-gmf2"
server_key: "8f612a72571f3d503cf1415fa42afaff"
server_key_iv: "3198d433dbfb688998d3ec455ca4c12a"
server_key_version: "encryptionkey@0"
I20260621 06:54:59.143337 2207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:54:59.161850 2207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:54:59.166383 2207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:54:59.166700 2207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:54:59.185628 2207 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:39383
I20260621 06:54:59.185653 2273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:39383 every 8 connection(s)
I20260621 06:54:59.187088 2207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:54:59.189682 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 2207
I20260621 06:54:59.189880 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:54:59.190506 20963 external_mini_cluster.cc:1428] Setting key a54b00587d35177a16db6b758e00d0d5
I20260621 06:54:59.190881 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:54:59.205564 2274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7: Bootstrap starting.
Jun 21 06:54:59 dist-test-slave-gmf2 krb5kdc[2194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:54:59.208889 2274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7: Neither blocks nor log segments found. Creating new log.
I20260621 06:54:59.210162 2274 log.cc:824] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7: Log is configured to *not* fsync() on all Append() calls
I20260621 06:54:59.213007 2274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7: No bootstrap required, opened a new log
I20260621 06:54:59.215349 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:54:59.192251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.20.120.254:39383)
0621 06:54:59.193041 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0621 06:54:59.193052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:54:59.193103 (+ 51us) server_negotiation.cc:407] Connection header received
0621 06:54:59.195449 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:54:59.195473 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:54:59.195874 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:54:59.196218 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:54:59.198157 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:59.199425 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:59.201332 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:54:59.202114 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:54:59.202248 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:54:59.207285 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:54:59.207310 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:54:59.207325 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:54:59.207359 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:54:59.210901 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:59.211737 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:59.211754 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:59.211760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:59.211936 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:54:59.212370 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:54:59.212375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:54:59.212385 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:54:59.212911 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:54:59.213266 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0621 06:54:59.213799 (+ 533us) server_negotiation.cc:299] Negotiation successful
0621 06:54:59.214258 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":369,"threads_started":1}
I20260621 06:54:59.219379 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:54:59.221086 2274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0355f94517a841c1a02e375741c41fa7" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39383 } }
I20260621 06:54:59.221496 2274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:54:59.221618 2274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0355f94517a841c1a02e375741c41fa7, State: Initialized, Role: FOLLOWER
I20260621 06:54:59.222203 2274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0355f94517a841c1a02e375741c41fa7" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39383 } }
I20260621 06:54:59.222456 2274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:54:59.222597 2274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:54:59.222779 2274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:54:59.224887 2274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0355f94517a841c1a02e375741c41fa7" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39383 } }
I20260621 06:54:59.225365 2274 leader_election.cc:302] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0355f94517a841c1a02e375741c41fa7; no voters:
I20260621 06:54:59.226007 2274 leader_election.cc:288] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:54:59.226167 2279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:54:59.226929 2279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [term 1 LEADER]: Becoming Leader. State: Replica: 0355f94517a841c1a02e375741c41fa7, State: Running, Role: LEADER
I20260621 06:54:59.227429 2279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0355f94517a841c1a02e375741c41fa7" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39383 } }
I20260621 06:54:59.228139 2274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:54:59.230472 2281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0355f94517a841c1a02e375741c41fa7. Latest consensus state: current_term: 1 leader_uuid: "0355f94517a841c1a02e375741c41fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0355f94517a841c1a02e375741c41fa7" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39383 } } }
I20260621 06:54:59.230909 2281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [sys.catalog]: This master's current role is: LEADER
I20260621 06:54:59.231443 2280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0355f94517a841c1a02e375741c41fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0355f94517a841c1a02e375741c41fa7" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 39383 } } }
I20260621 06:54:59.231782 2280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7 [sys.catalog]: This master's current role is: LEADER
I20260621 06:54:59.234418 2286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:54:59.239475 2286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:54:59.248312 2286 catalog_manager.cc:1347] Generated new cluster ID: 8438693c2b46433bbe6b4debcc975033
I20260621 06:54:59.248437 2286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:54:59.256981 2286 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:54:59.258297 2286 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:54:59.270066 2286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0355f94517a841c1a02e375741c41fa7: Generated new TSK 0
I20260621 06:54:59.270872 2286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:54:59.280056 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 2207
I20260621 06:54:59.281277 2269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:54:59.370312 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 2207
2026-06-21T06:54:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1889 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:54:59 dist-test-slave-gmf2 krb5kdc[2302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:54:59 dist-test-slave-gmf2 krb5kdc[2302](info): set up 2 sockets
Jun 21 06:54:59 dist-test-slave-gmf2 krb5kdc[2302](info): commencing operation
krb5kdc: starting...
W20260621 06:55:00.462448 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:55:00 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:55:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:55:00.699828 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34489
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:34489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:55:01.036307 2315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:01.036628 2315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:01.036835 2315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:01.044170 2315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:55:01.044281 2315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:01.044399 2315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:01.044512 2315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:55:01.044601 2315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:55:01.054777 2315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:34489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:01.056111 2315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:01.057443 2315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:01.069038 2320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:01.068938 2323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:01.068984 2321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:01.069972 2315 server_base.cc:1034] running on GCE node
I20260621 06:55:01.070746 2315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:01.072206 2315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:01.073434 2315 hybrid_clock.cc:648] HybridClock initialized: now 1782024901073388 us; error 53 us; skew 500 ppm
I20260621 06:55:01.073786 2315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:01 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024901, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:01.076939 2315 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:55:01.078467 2315 webserver.cc:460] Webserver started at http://127.20.120.254:45761/ using document root <none> and password file <none>
I20260621 06:55:01.079138 2315 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:01.079291 2315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:01.079628 2315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:55:01.082768 2315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "3a787ded34fd42a09ad258378b7144cc"
format_stamp: "Formatted at 2026-06-21 06:55:01 on dist-test-slave-gmf2"
server_key: "3de60edd34f2447a22597f9717604406"
server_key_iv: "1f474d68e94da5d6ef16a67d5620ca4a"
server_key_version: "encryptionkey@0"
I20260621 06:55:01.083530 2315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "3a787ded34fd42a09ad258378b7144cc"
format_stamp: "Formatted at 2026-06-21 06:55:01 on dist-test-slave-gmf2"
server_key: "3de60edd34f2447a22597f9717604406"
server_key_iv: "1f474d68e94da5d6ef16a67d5620ca4a"
server_key_version: "encryptionkey@0"
I20260621 06:55:01.088557 2315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:55:01.091956 2330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:01.093273 2315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 06:55:01.093451 2315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "3a787ded34fd42a09ad258378b7144cc"
format_stamp: "Formatted at 2026-06-21 06:55:01 on dist-test-slave-gmf2"
server_key: "3de60edd34f2447a22597f9717604406"
server_key_iv: "1f474d68e94da5d6ef16a67d5620ca4a"
server_key_version: "encryptionkey@0"
I20260621 06:55:01.093655 2315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:01.108318 2315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:01.112785 2315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:01.113077 2315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:01.131176 2315 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:34489
I20260621 06:55:01.131214 2381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:34489 every 8 connection(s)
I20260621 06:55:01.132567 2315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:55:01.136250 2382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:01.142936 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 2315
I20260621 06:55:01.143160 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:55:01.144003 20963 external_mini_cluster.cc:1428] Setting key 17cc24f71ed86e50087355bd3d4a6e2c
I20260621 06:55:01.147100 2382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Bootstrap starting.
I20260621 06:55:01.150666 2382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:01.151844 2382 log.cc:824] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:01.154663 2382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: No bootstrap required, opened a new log
Jun 21 06:55:01 dist-test-slave-gmf2 krb5kdc[2302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:01.161986 2382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } }
I20260621 06:55:01.162370 2382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:01.162474 2382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a787ded34fd42a09ad258378b7144cc, State: Initialized, Role: FOLLOWER
I20260621 06:55:01.163162 2382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } }
I20260621 06:55:01.163476 2382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:55:01.163632 2382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:55:01.163818 2382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:01.166335 2382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } }
I20260621 06:55:01.166549 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:01.146031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.20.120.254:34489)
0621 06:55:01.146773 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:01.146785 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:01.146824 (+ 39us) server_negotiation.cc:407] Connection header received
0621 06:55:01.148529 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:01.148551 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:01.148940 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:01.149264 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:01.151154 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:01.152503 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:01.153852 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:01.154570 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:01.154773 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:01.158804 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:01.158812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:01.158830 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:01.158863 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:01.162003 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:01.162893 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:01.162901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:01.162909 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:01.163108 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:01.163553 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:01.163557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:01.163559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:01.164212 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:01.164607 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:01.165074 (+ 467us) server_negotiation.cc:299] Negotiation successful
0621 06:55:01.165499 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":369,"threads_started":1}
I20260621 06:55:01.166925 2382 leader_election.cc:302] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a787ded34fd42a09ad258378b7144cc; no voters:
I20260621 06:55:01.167634 2382 leader_election.cc:288] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:55:01.167774 2387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:55:01.168529 2387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 1 LEADER]: Becoming Leader. State: Replica: 3a787ded34fd42a09ad258378b7144cc, State: Running, Role: LEADER
I20260621 06:55:01.169155 2387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } }
I20260621 06:55:01.171656 2382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:55:01.172236 2388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a787ded34fd42a09ad258378b7144cc. Latest consensus state: current_term: 1 leader_uuid: "3a787ded34fd42a09ad258378b7144cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } } }
I20260621 06:55:01.172266 2389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a787ded34fd42a09ad258378b7144cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } } }
I20260621 06:55:01.172626 2388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: This master's current role is: LEADER
I20260621 06:55:01.173259 2389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: This master's current role is: LEADER
I20260621 06:55:01.176868 2392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:55:01.181936 2392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:55:01.191489 2392 catalog_manager.cc:1347] Generated new cluster ID: ac1eb1f3191f4f69b5fc38f4b086adba
I20260621 06:55:01.191586 2392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:55:01.228909 2392 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:55:01.230177 2392 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:55:01.239426 2392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Generated new TSK 0
I20260621 06:55:01.240429 2392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:55:01.295461 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34489
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:55:01.643783 2408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:01.644074 2408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:01.644169 2408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:01.651592 2408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:01.651719 2408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:01.651836 2408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:55:01.661705 2408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:01.663041 2408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:01.664319 2408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:01.678205 2413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:01.678457 2408 server_base.cc:1034] running on GCE node
W20260621 06:55:01.678115 2414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:01.678143 2416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:01.679476 2408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:01.680328 2408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:01.681524 2408 hybrid_clock.cc:648] HybridClock initialized: now 1782024901681461 us; error 54 us; skew 500 ppm
I20260621 06:55:01.681838 2408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:01 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024901, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:01.685047 2408 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:55:01.686544 2408 webserver.cc:460] Webserver started at http://127.20.120.193:46275/ using document root <none> and password file <none>
I20260621 06:55:01.687235 2408 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:01.687369 2408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:01.687685 2408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:55:01.690831 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "93a0a7df2c69447eb6ba3aa1564b3b02"
format_stamp: "Formatted at 2026-06-21 06:55:01 on dist-test-slave-gmf2"
server_key: "cd720c3475dbd21bcd9d7b099e08dac4"
server_key_iv: "e99fcca2a65a69a8334afac08f5beab4"
server_key_version: "encryptionkey@0"
I20260621 06:55:01.691645 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "93a0a7df2c69447eb6ba3aa1564b3b02"
format_stamp: "Formatted at 2026-06-21 06:55:01 on dist-test-slave-gmf2"
server_key: "cd720c3475dbd21bcd9d7b099e08dac4"
server_key_iv: "e99fcca2a65a69a8334afac08f5beab4"
server_key_version: "encryptionkey@0"
I20260621 06:55:01.696960 2408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 06:55:01.700508 2423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:01.701841 2408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:55:01.702057 2408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "93a0a7df2c69447eb6ba3aa1564b3b02"
format_stamp: "Formatted at 2026-06-21 06:55:01 on dist-test-slave-gmf2"
server_key: "cd720c3475dbd21bcd9d7b099e08dac4"
server_key_iv: "e99fcca2a65a69a8334afac08f5beab4"
server_key_version: "encryptionkey@0"
I20260621 06:55:01.702280 2408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:01.721635 2408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:01.725600 2408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:01.725898 2408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:01.726948 2408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:01.728324 2408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:01.728428 2408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:01.728585 2408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:01.728654 2408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:01.762210 2408 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:45313
I20260621 06:55:01.762269 2535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:45313 every 8 connection(s)
I20260621 06:55:01.763652 2408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:55:01.768050 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 2408
I20260621 06:55:01.768276 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:55:01.769167 20963 external_mini_cluster.cc:1428] Setting key e758261e5ff1f831e7b75123b422f0ee
Jun 21 06:55:01 dist-test-slave-gmf2 krb5kdc[2302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024901, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:01.789279 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:01.766660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:48189 (local address 127.20.120.254:34489)
0621 06:55:01.767136 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:01.767141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:01.768412 (+ 1271us) server_negotiation.cc:407] Connection header received
0621 06:55:01.771004 (+ 2592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:01.771009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:01.771101 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:01.771238 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:01.773537 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:01.774373 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:01.777016 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:01.777450 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:01.777555 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:01.782200 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:01.782210 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:01.782219 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:01.782267 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:01.785799 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:01.786618 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:01.786626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:01.786632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:01.786787 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:01.787229 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:01.787236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:01.787243 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:01.787649 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:01.787821 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:01.788526 (+ 705us) server_negotiation.cc:299] Negotiation successful
0621 06:55:01.788789 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":246,"threads_started":1}
I20260621 06:55:01.790030 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:01.767279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:48189)
0621 06:55:01.768211 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:01.768277 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:01.770711 (+ 2434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:01.771546 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:01.771558 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:01.772396 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:55:01.773329 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:01.773361 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:01.774566 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:01.774574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:01.776842 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:01.776862 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:01.777707 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:01.777716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:01.778109 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:01.778959 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:55:01.778999 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:55:01.781866 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:55:01.786037 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:01.786052 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:01.786067 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:01.786486 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:01.786900 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:01.786903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:01.786905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:01.787096 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:01.787772 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:55:01.787778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:55:01.788122 (+ 344us) client_negotiation.cc:769] Sending connection context
0621 06:55:01.788358 (+ 236us) client_negotiation.cc:240] Negotiation successful
0621 06:55:01.788780 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":343,"threads_started":1}
I20260621 06:55:01.791826 2536 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34489
I20260621 06:55:01.792172 2536 heartbeater.cc:464] Registering TS with master...
I20260621 06:55:01.792959 2536 heartbeater.cc:511] Master 127.20.120.254:34489 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
I20260621 06:55:01.795297 2347 ts_manager.cc:194] Registered new tserver with Master: 93a0a7df2c69447eb6ba3aa1564b3b02 (127.20.120.193:45313)
I20260621 06:55:01.796977 2347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:48189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:55:01.826908 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34489
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:55:02.150985 2542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:02.151247 2542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:02.151342 2542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:02.158476 2542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:02.158582 2542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:02.158804 2542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:55:02.168470 2542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:02.169946 2542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:02.171353 2542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:02.184362 2547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:02.184417 2548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:02.184374 2550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:02.185461 2542 server_base.cc:1034] running on GCE node
I20260621 06:55:02.186136 2542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:02.186931 2542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:02.188160 2542 hybrid_clock.cc:648] HybridClock initialized: now 1782024902188122 us; error 58 us; skew 500 ppm
I20260621 06:55:02.188472 2542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:02 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024902, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:02.191707 2542 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:55:02.193075 2542 webserver.cc:460] Webserver started at http://127.20.120.194:37267/ using document root <none> and password file <none>
I20260621 06:55:02.193709 2542 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:02.193805 2542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:02.194120 2542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:55:02.196856 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "c74b10b35ad74dfa9a6965b863724d97"
format_stamp: "Formatted at 2026-06-21 06:55:02 on dist-test-slave-gmf2"
server_key: "ca8a642ffc594bfbdf69e8166d5a795e"
server_key_iv: "4d036dbde4639689833d32d563880a70"
server_key_version: "encryptionkey@0"
I20260621 06:55:02.197562 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "c74b10b35ad74dfa9a6965b863724d97"
format_stamp: "Formatted at 2026-06-21 06:55:02 on dist-test-slave-gmf2"
server_key: "ca8a642ffc594bfbdf69e8166d5a795e"
server_key_iv: "4d036dbde4639689833d32d563880a70"
server_key_version: "encryptionkey@0"
I20260621 06:55:02.202373 2542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 06:55:02.205622 2557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:02.206979 2542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:55:02.207103 2542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "c74b10b35ad74dfa9a6965b863724d97"
format_stamp: "Formatted at 2026-06-21 06:55:02 on dist-test-slave-gmf2"
server_key: "ca8a642ffc594bfbdf69e8166d5a795e"
server_key_iv: "4d036dbde4639689833d32d563880a70"
server_key_version: "encryptionkey@0"
I20260621 06:55:02.207265 2542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:02.221392 2542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:02.225744 2542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:02.226130 2542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:02.227105 2542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:02.228458 2542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:02.228538 2542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:02.228637 2542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:02.228688 2542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:02.264147 2542 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:46049
I20260621 06:55:02.264170 2669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:46049 every 8 connection(s)
I20260621 06:55:02.265561 2542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:55:02.269333 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 2542
I20260621 06:55:02.269549 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:55:02.270165 20963 external_mini_cluster.cc:1428] Setting key e0a04e05d67361d1f543c23c47705374
Jun 21 06:55:02 dist-test-slave-gmf2 krb5kdc[2302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024902, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:02.289985 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:02.268254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:35825 (local address 127.20.120.254:34489)
0621 06:55:02.268474 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:02.268478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:02.270234 (+ 1756us) server_negotiation.cc:407] Connection header received
0621 06:55:02.272434 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:02.272441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:02.272530 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:02.272647 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:02.274870 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.275586 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.278128 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.278616 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.278716 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:02.282963 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:02.282972 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:02.282979 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:02.283032 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:02.286717 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:02.287457 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:02.287465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:02.287472 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:02.287630 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:02.288057 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:02.288065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:02.288071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:02.288463 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:02.288621 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:02.289288 (+ 667us) server_negotiation.cc:299] Negotiation successful
0621 06:55:02.289534 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 06:55:02.290779 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:02.268779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:35825)
0621 06:55:02.270054 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:02.270106 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:02.272123 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:02.272981 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:02.272992 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:02.273805 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:55:02.274696 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:02.274724 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.275736 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.275744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:02.277994 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:02.278012 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.278851 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.278860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:02.279216 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:02.280001 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:55:02.280036 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:55:02.282711 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:55:02.286862 (+ 4151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:02.286869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:02.286883 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:02.287327 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:02.287738 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:02.287741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:02.287743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:02.287922 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:02.288581 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:55:02.288587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:55:02.288896 (+ 309us) client_negotiation.cc:769] Sending connection context
0621 06:55:02.289122 (+ 226us) client_negotiation.cc:240] Negotiation successful
0621 06:55:02.289514 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"thread_start_us":751,"threads_started":1}
I20260621 06:55:02.292450 2670 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34489
I20260621 06:55:02.292807 2670 heartbeater.cc:464] Registering TS with master...
I20260621 06:55:02.293814 2670 heartbeater.cc:511] Master 127.20.120.254:34489 requested a full tablet report, sending...
I20260621 06:55:02.295521 2347 ts_manager.cc:194] Registered new tserver with Master: c74b10b35ad74dfa9a6965b863724d97 (127.20.120.194:46049)
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
I20260621 06:55:02.296408 2347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:35825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:55:02.325040 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34489
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:55:02.654798 2675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:02.655088 2675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:02.655184 2675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:02.662312 2675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:02.662427 2675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:02.662539 2675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:55:02.672394 2675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:02.673784 2675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:02.675163 2675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:02.688395 2680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:02.688485 2681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:02.688441 2683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:02.689452 2675 server_base.cc:1034] running on GCE node
I20260621 06:55:02.690218 2675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:02.691226 2675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:02.692457 2675 hybrid_clock.cc:648] HybridClock initialized: now 1782024902692415 us; error 51 us; skew 500 ppm
I20260621 06:55:02.692777 2675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:02 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024902, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:02.695959 2675 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:55:02.697449 2675 webserver.cc:460] Webserver started at http://127.20.120.195:43947/ using document root <none> and password file <none>
I20260621 06:55:02.698244 2675 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:02.698382 2675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:02.698691 2675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:55:02.701488 2675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "1e4c49a9fe4b419181608e0a84c48682"
format_stamp: "Formatted at 2026-06-21 06:55:02 on dist-test-slave-gmf2"
server_key: "d2022938de9db6ab3b15769ffa2fbdbb"
server_key_iv: "29b72a37cf3b9e100198e4c235b43b38"
server_key_version: "encryptionkey@0"
I20260621 06:55:02.702317 2675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "1e4c49a9fe4b419181608e0a84c48682"
format_stamp: "Formatted at 2026-06-21 06:55:02 on dist-test-slave-gmf2"
server_key: "d2022938de9db6ab3b15769ffa2fbdbb"
server_key_iv: "29b72a37cf3b9e100198e4c235b43b38"
server_key_version: "encryptionkey@0"
I20260621 06:55:02.707739 2675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:55:02.711172 2690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:02.712441 2675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:55:02.712607 2675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "1e4c49a9fe4b419181608e0a84c48682"
format_stamp: "Formatted at 2026-06-21 06:55:02 on dist-test-slave-gmf2"
server_key: "d2022938de9db6ab3b15769ffa2fbdbb"
server_key_iv: "29b72a37cf3b9e100198e4c235b43b38"
server_key_version: "encryptionkey@0"
I20260621 06:55:02.712787 2675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:02.744153 2675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:02.748239 2675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:02.748526 2675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:02.749524 2675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:02.750927 2675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:02.751034 2675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:02.751195 2675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:02.751274 2675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:02.784749 2675 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:32887
I20260621 06:55:02.784790 2802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:32887 every 8 connection(s)
I20260621 06:55:02.786026 2675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:55:02.788450 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 2675
I20260621 06:55:02.788641 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:55:02.789059 20963 external_mini_cluster.cc:1428] Setting key f8280312f4b79c81113f5cb5d0059791
Jun 21 06:55:02 dist-test-slave-gmf2 krb5kdc[2302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024902, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:02.800920 2536 heartbeater.cc:503] Master 127.20.120.254:34489 was elected leader, sending a full tablet report...
I20260621 06:55:02.805872 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:02.788655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:42809 (local address 127.20.120.254:34489)
0621 06:55:02.788839 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:02.788845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:02.790159 (+ 1314us) server_negotiation.cc:407] Connection header received
0621 06:55:02.792120 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:02.792125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:02.792190 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:02.792307 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:02.793679 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.794556 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.795947 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.796336 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.796424 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:02.799742 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:02.799746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:02.799749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:02.799776 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:02.802703 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:02.803568 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:02.803573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:02.803575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:02.803763 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:02.804268 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:02.804271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:02.804273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:02.804655 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:02.804812 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:02.805504 (+ 692us) server_negotiation.cc:299] Negotiation successful
0621 06:55:02.805652 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 06:55:02.807026 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:02.789151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:42809)
0621 06:55:02.790003 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:02.790046 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:02.791845 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:02.792507 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:02.792515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:02.792962 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:55:02.793532 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:02.793549 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.794691 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.794694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:02.795835 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:02.795846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.796512 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.796517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:02.796704 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:02.797490 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:55:02.797512 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:55:02.799571 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:55:02.802838 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:02.802846 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:02.802861 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:02.803451 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:02.803876 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:02.803881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:02.803884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:02.804150 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:02.804797 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:55:02.804802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:55:02.805179 (+ 377us) client_negotiation.cc:769] Sending connection context
0621 06:55:02.805418 (+ 239us) client_negotiation.cc:240] Negotiation successful
0621 06:55:02.805833 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":371,"threads_started":1}
I20260621 06:55:02.808315 2803 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34489
I20260621 06:55:02.808591 2803 heartbeater.cc:464] Registering TS with master...
I20260621 06:55:02.809249 2803 heartbeater.cc:511] Master 127.20.120.254:34489 requested a full tablet report, sending...
I20260621 06:55:02.810642 2347 ts_manager.cc:194] Registered new tserver with Master: 1e4c49a9fe4b419181608e0a84c48682 (127.20.120.195:32887)
I20260621 06:55:02.811440 2347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:42809
I20260621 06:55:02.817093 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:55:02.833618 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:02.821534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.20.120.254:34489)
0621 06:55:02.821769 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:02.821775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:02.822104 (+ 329us) server_negotiation.cc:407] Connection header received
0621 06:55:02.822245 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:02.822249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:02.822313 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:02.822392 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:02.823758 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.824479 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.825769 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.826254 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.826346 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:02.828535 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:02.828539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:02.828542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:02.828572 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:02.831174 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:02.831878 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:02.831882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:02.831884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:02.832017 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:02.832325 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:02.832328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:02.832330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:02.832621 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:02.832824 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:02.833083 (+ 259us) server_negotiation.cc:299] Negotiation successful
0621 06:55:02.833268 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 06:55:02.837700 2347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:55:02.844751 2347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:55:02.869043 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:02.858732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.20.120.194:46049)
0621 06:55:02.859383 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:02.859392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:02.860010 (+ 618us) server_negotiation.cc:407] Connection header received
0621 06:55:02.860247 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:02.860258 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:02.860548 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:02.861557 (+ 1009us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:02.861764 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.863159 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.866076 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.867576 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.867740 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:02.867918 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:02.868513 (+ 595us) server_negotiation.cc:299] Negotiation successful
0621 06:55:02.868685 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":663,"threads_started":1}
I20260621 06:55:02.869760 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:02.860145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:34894)
0621 06:55:02.860970 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:02.860989 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:02.861111 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:02.862635 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:02.862638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:02.862660 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:02.863053 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:02.863064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.864559 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.864567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:02.866709 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:02.866727 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.868478 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.868484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:02.868630 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:02.868661 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 06:55:02.868725 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 06:55:02.868854 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":69,"thread_start_us":803,"threads_started":1}
I20260621 06:55:02.870088 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:02.860785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34894 (local address 127.20.120.193:45313)
0621 06:55:02.861729 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:02.861738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:02.861764 (+ 26us) server_negotiation.cc:407] Connection header received
0621 06:55:02.861864 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:02.861872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:02.862132 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:02.862330 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:02.863244 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.864385 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.866907 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.868252 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.869102 (+ 850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:02.869370 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:02.869436 (+ 66us) server_negotiation.cc:299] Negotiation successful
0621 06:55:02.869594 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":630,"threads_started":1}
I20260621 06:55:02.870119 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:02.858104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:46049 (local address 127.0.0.1:41140)
0621 06:55:02.859888 (+ 1784us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:02.859903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:02.860058 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:02.860817 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:02.860826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:02.860861 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:02.861365 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:02.861379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.863298 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.863303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:02.865888 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:02.865931 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.867783 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.867789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:02.867976 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:02.868032 (+ 56us) client_negotiation.cc:769] Sending connection context
0621 06:55:02.868461 (+ 429us) client_negotiation.cc:240] Negotiation successful
0621 06:55:02.868677 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1687,"mutex_wait_us":4,"thread_start_us":1578,"threads_started":1}
I20260621 06:55:02.870147 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:02.857578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:32887 (local address 127.0.0.1:34846)
0621 06:55:02.858517 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:02.858568 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:02.858755 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:02.862337 (+ 3582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:02.862342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:02.862368 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:02.862779 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:02.862795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.864810 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.864815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:02.866857 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:02.866871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.868258 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.868264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:02.869002 (+ 738us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:02.869033 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 06:55:02.869113 (+ 80us) client_negotiation.cc:240] Negotiation successful
0621 06:55:02.869617 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":1320,"threads_started":1}
I20260621 06:55:02.871281 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:02.857882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34846 (local address 127.20.120.195:32887)
0621 06:55:02.861806 (+ 3924us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:02.861813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:02.861840 (+ 27us) server_negotiation.cc:407] Connection header received
0621 06:55:02.861947 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:02.861954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:02.862141 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:02.862342 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:02.863005 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.864640 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.867026 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:02.868046 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:02.868239 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:02.868402 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:02.869373 (+ 971us) server_negotiation.cc:299] Negotiation successful
0621 06:55:02.869539 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3762,"thread_start_us":2424,"threads_started":1}
I20260621 06:55:02.874372 2471 tablet_service.cc:1467] Processing CreateTablet for tablet cbbde6bf5e4d45d48d4781181224a146 (DEFAULT_TABLE table=test-table [id=c08601dd831b47f7b91a4d624815180d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:55:02.874372 2605 tablet_service.cc:1467] Processing CreateTablet for tablet cbbde6bf5e4d45d48d4781181224a146 (DEFAULT_TABLE table=test-table [id=c08601dd831b47f7b91a4d624815180d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:55:02.874720 2738 tablet_service.cc:1467] Processing CreateTablet for tablet cbbde6bf5e4d45d48d4781181224a146 (DEFAULT_TABLE table=test-table [id=c08601dd831b47f7b91a4d624815180d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:55:02.875849 2605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbbde6bf5e4d45d48d4781181224a146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:02.875844 2471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbbde6bf5e4d45d48d4781181224a146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:02.876000 2738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbbde6bf5e4d45d48d4781181224a146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:02.887681 2821 tablet_bootstrap.cc:492] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682: Bootstrap starting.
I20260621 06:55:02.890470 2822 tablet_bootstrap.cc:492] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97: Bootstrap starting.
I20260621 06:55:02.893057 2821 tablet_bootstrap.cc:654] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:02.894037 2823 tablet_bootstrap.cc:492] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: Bootstrap starting.
I20260621 06:55:02.894285 2822 tablet_bootstrap.cc:654] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:02.895064 2821 log.cc:824] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:02.895524 2822 log.cc:824] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:02.897382 2823 tablet_bootstrap.cc:654] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:02.898597 2821 tablet_bootstrap.cc:492] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682: No bootstrap required, opened a new log
I20260621 06:55:02.898739 2822 tablet_bootstrap.cc:492] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97: No bootstrap required, opened a new log
I20260621 06:55:02.898721 2823 log.cc:824] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:02.898913 2821 ts_tablet_manager.cc:1397] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260621 06:55:02.898998 2822 ts_tablet_manager.cc:1397] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 06:55:02.901690 2823 tablet_bootstrap.cc:492] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: No bootstrap required, opened a new log
I20260621 06:55:02.901979 2823 ts_tablet_manager.cc:1397] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260621 06:55:02.906533 2821 raft_consensus.cc:348] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:02.906884 2821 raft_consensus.cc:374] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:02.906981 2821 raft_consensus.cc:729] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4c49a9fe4b419181608e0a84c48682, State: Initialized, Role: FOLLOWER
I20260621 06:55:02.907577 2821 consensus_queue.cc:260] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:02.909217 2822 raft_consensus.cc:348] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:02.909621 2822 raft_consensus.cc:374] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:02.909709 2822 raft_consensus.cc:729] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74b10b35ad74dfa9a6965b863724d97, State: Initialized, Role: FOLLOWER
I20260621 06:55:02.910526 2803 heartbeater.cc:503] Master 127.20.120.254:34489 was elected leader, sending a full tablet report...
I20260621 06:55:02.910393 2822 consensus_queue.cc:260] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:02.911191 2821 ts_tablet_manager.cc:1428] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260621 06:55:02.913106 2823 raft_consensus.cc:348] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:02.913676 2823 raft_consensus.cc:374] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:02.913796 2823 raft_consensus.cc:729] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a0a7df2c69447eb6ba3aa1564b3b02, State: Initialized, Role: FOLLOWER
I20260621 06:55:02.914525 2670 heartbeater.cc:503] Master 127.20.120.254:34489 was elected leader, sending a full tablet report...
I20260621 06:55:02.914482 2823 consensus_queue.cc:260] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:02.916256 2823 ts_tablet_manager.cc:1428] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260621 06:55:02.917361 2822 ts_tablet_manager.cc:1428] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
W20260621 06:55:03.016152 2537 tablet.cc:2369] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:55:03.017645 2671 tablet.cc:2369] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:55:03.037649 2804 tablet.cc:2369] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:55:03.197309 2829 raft_consensus.cc:482] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:55:03.197618 2829 raft_consensus.cc:504] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:03.199314 2829 leader_election.cc:288] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e4c49a9fe4b419181608e0a84c48682 (127.20.120.195:32887), c74b10b35ad74dfa9a6965b863724d97 (127.20.120.194:46049)
I20260621 06:55:03.205655 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.199751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:59065 (local address 127.20.120.194:46049)
0621 06:55:03.200012 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:03.200022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:03.200557 (+ 535us) server_negotiation.cc:407] Connection header received
0621 06:55:03.200774 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:03.200778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:03.200836 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:03.200928 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:03.201478 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.202214 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.204028 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.204791 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.204959 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:03.205037 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:03.205279 (+ 242us) server_negotiation.cc:299] Negotiation successful
0621 06:55:03.205382 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:55:03.205771 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.199774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:42709 (local address 127.20.120.195:32887)
0621 06:55:03.200178 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:03.200186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:03.200685 (+ 499us) server_negotiation.cc:407] Connection header received
0621 06:55:03.200873 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:03.200876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:03.200929 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:03.201027 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:03.201570 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.202332 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.204086 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.204890 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.205065 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:03.205126 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:03.205401 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 06:55:03.205524 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260621 06:55:03.206106 2830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.199619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:46049 (local address 127.20.120.193:59065)
0621 06:55:03.200430 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:03.200448 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:03.200615 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:03.200955 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:03.200959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:03.200982 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:03.201331 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:03.201343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.202347 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.202353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:03.203873 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:03.203885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.204988 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.204993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:03.205143 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:03.205163 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 06:55:03.205257 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 06:55:03.205366 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":33,"thread_start_us":381,"threads_started":1}
I20260621 06:55:03.206424 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.199710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:32887 (local address 127.20.120.193:42709)
0621 06:55:03.200592 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:03.200612 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:03.200746 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:03.201043 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:03.201046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:03.201082 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:03.201428 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:03.201435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.202462 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.202470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:03.203937 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:03.203948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.205086 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.205091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:03.205257 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:03.205286 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 06:55:03.205373 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 06:55:03.205476 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":41,"thread_start_us":600,"threads_started":1}
I20260621 06:55:03.207206 2625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbbde6bf5e4d45d48d4781181224a146" candidate_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74b10b35ad74dfa9a6965b863724d97" is_pre_election: true
I20260621 06:55:03.207228 2758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbbde6bf5e4d45d48d4781181224a146" candidate_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4c49a9fe4b419181608e0a84c48682" is_pre_election: true
I20260621 06:55:03.207674 2758 raft_consensus.cc:2454] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a0a7df2c69447eb6ba3aa1564b3b02 in term 0.
I20260621 06:55:03.207686 2625 raft_consensus.cc:2454] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a0a7df2c69447eb6ba3aa1564b3b02 in term 0.
I20260621 06:55:03.208266 2427 leader_election.cc:302] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a0a7df2c69447eb6ba3aa1564b3b02, c74b10b35ad74dfa9a6965b863724d97; no voters:
I20260621 06:55:03.208657 2829 raft_consensus.cc:2781] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:55:03.208817 2829 raft_consensus.cc:482] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:55:03.208925 2829 raft_consensus.cc:3037] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:03.211272 2829 raft_consensus.cc:504] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:03.211863 2829 leader_election.cc:288] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [CANDIDATE]: Term 1 election: Requested vote from peers 1e4c49a9fe4b419181608e0a84c48682 (127.20.120.195:32887), c74b10b35ad74dfa9a6965b863724d97 (127.20.120.194:46049)
I20260621 06:55:03.212234 2758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbbde6bf5e4d45d48d4781181224a146" candidate_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4c49a9fe4b419181608e0a84c48682"
I20260621 06:55:03.212339 2625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbbde6bf5e4d45d48d4781181224a146" candidate_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74b10b35ad74dfa9a6965b863724d97"
I20260621 06:55:03.212433 2758 raft_consensus.cc:3037] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:03.212528 2625 raft_consensus.cc:3037] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:03.215657 2758 raft_consensus.cc:2454] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a0a7df2c69447eb6ba3aa1564b3b02 in term 1.
I20260621 06:55:03.215657 2625 raft_consensus.cc:2454] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a0a7df2c69447eb6ba3aa1564b3b02 in term 1.
I20260621 06:55:03.216054 2424 leader_election.cc:302] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e4c49a9fe4b419181608e0a84c48682, 93a0a7df2c69447eb6ba3aa1564b3b02, c74b10b35ad74dfa9a6965b863724d97; no voters:
I20260621 06:55:03.216383 2829 raft_consensus.cc:2781] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:55:03.216831 2829 raft_consensus.cc:686] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 1 LEADER]: Becoming Leader. State: Replica: 93a0a7df2c69447eb6ba3aa1564b3b02, State: Running, Role: LEADER
I20260621 06:55:03.217356 2829 consensus_queue.cc:237] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:03.221832 2347 catalog_manager.cc:5495] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93a0a7df2c69447eb6ba3aa1564b3b02 (127.20.120.193). New cstate: current_term: 1 leader_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } health_report { overall_health: HEALTHY } } }
I20260621 06:55:03.324522 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.318986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.20.120.193:45313)
0621 06:55:03.319193 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:03.319200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:03.319220 (+ 20us) server_negotiation.cc:407] Connection header received
0621 06:55:03.319487 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:03.319511 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:03.319624 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:03.319778 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:55:03.320216 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.320893 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.322351 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.322781 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.322923 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:03.323435 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:55:03.323935 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:55:03.324099 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:03.324179 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 06:55:03.324256 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 06:55:03.332446 2758 raft_consensus.cc:1261] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682 [term 1 FOLLOWER]: Refusing update from remote peer 93a0a7df2c69447eb6ba3aa1564b3b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:55:03.332445 2625 raft_consensus.cc:1261] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97 [term 1 FOLLOWER]: Refusing update from remote peer 93a0a7df2c69447eb6ba3aa1564b3b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:55:03.333428 2832 consensus_queue.cc:1035] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:55:03.333727 2829 consensus_queue.cc:1035] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:55:03.345305 2836 mvcc.cc:205] Tried to move back new op lower bound from 7299174004040433664 to 7299174003582124032. Current Snapshot: MvccSnapshot[applied={T|T < 7299174004040433664}]
I20260621 06:55:03.349233 2837 mvcc.cc:205] Tried to move back new op lower bound from 7299174004040433664 to 7299174003582124032. Current Snapshot: MvccSnapshot[applied={T|T < 7299174004040433664}]
I20260621 06:55:03.357308 2346 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:55:03.357642 2346 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:55:03.361238 2346 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Sending DeleteTablet for 3 replicas of tablet cbbde6bf5e4d45d48d4781181224a146
I20260621 06:55:03.362392 2738 tablet_service.cc:1514] Processing DeleteTablet for tablet cbbde6bf5e4d45d48d4781181224a146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:55:03 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:34846
I20260621 06:55:03.362552 2605 tablet_service.cc:1514] Processing DeleteTablet for tablet cbbde6bf5e4d45d48d4781181224a146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:55:03 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:41140
I20260621 06:55:03.362695 2471 tablet_service.cc:1514] Processing DeleteTablet for tablet cbbde6bf5e4d45d48d4781181224a146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:55:03 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:34894
I20260621 06:55:03.363469 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 2408
I20260621 06:55:03.380694 2846 tablet_replica.cc:330] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97: stopping tablet replica
I20260621 06:55:03.381314 2846 raft_consensus.cc:2229] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:55:03.382026 2846 raft_consensus.cc:2258] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:55:03.382048 2845 tablet_replica.cc:330] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682: stopping tablet replica
I20260621 06:55:03.382418 2845 raft_consensus.cc:2229] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:55:03.382886 2845 raft_consensus.cc:2258] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:55:03.384479 2846 ts_tablet_manager.cc:1905] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:03.386451 2845 ts_tablet_manager.cc:1905] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:03.391517 2846 ts_tablet_manager.cc:1918] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:55:03.391762 2846 log.cc:1196] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/cbbde6bf5e4d45d48d4781181224a146
I20260621 06:55:03.392313 2846 ts_tablet_manager.cc:1939] T cbbde6bf5e4d45d48d4781181224a146 P c74b10b35ad74dfa9a6965b863724d97: Deleting consensus metadata
Jun 21 06:55:03 dist-test-slave-gmf2 krb5kdc[2302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:55:03.393021 2845 ts_tablet_manager.cc:1918] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:55:03.393203 2845 log.cc:1196] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/cbbde6bf5e4d45d48d4781181224a146
I20260621 06:55:03.393682 2847 tablet_replica.cc:330] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: stopping tablet replica
I20260621 06:55:03.393699 2845 ts_tablet_manager.cc:1939] T cbbde6bf5e4d45d48d4781181224a146 P 1e4c49a9fe4b419181608e0a84c48682: Deleting consensus metadata
I20260621 06:55:03.394132 2847 raft_consensus.cc:2229] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:55:03.394539 2334 catalog_manager.cc:4841] TS c74b10b35ad74dfa9a6965b863724d97 (127.20.120.194:46049): tablet cbbde6bf5e4d45d48d4781181224a146 (table test-table [id=c08601dd831b47f7b91a4d624815180d]) successfully deleted
I20260621 06:55:03.395344 2331 catalog_manager.cc:4841] TS 1e4c49a9fe4b419181608e0a84c48682 (127.20.120.195:32887): tablet cbbde6bf5e4d45d48d4781181224a146 (table test-table [id=c08601dd831b47f7b91a4d624815180d]) successfully deleted
I20260621 06:55:03.395715 2847 raft_consensus.cc:2258] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:55:03.397557 2847 ts_tablet_manager.cc:1905] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:03.401290 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.364784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.20.120.193:45313)
0621 06:55:03.365012 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:03.365021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:03.372439 (+ 7418us) server_negotiation.cc:407] Connection header received
0621 06:55:03.373364 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:03.373373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:03.373485 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:03.373584 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:03.378483 (+ 4899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.380108 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.384799 (+ 4691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.386937 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.387103 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:03.394539 (+ 7436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:03.394562 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:03.394579 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:03.394633 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:03.398401 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:03.399075 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:03.399083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:03.399086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:03.399236 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:03.399632 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:03.399640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:03.399643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:03.400007 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:03.400327 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:03.400592 (+ 265us) server_negotiation.cc:299] Negotiation successful
0621 06:55:03.400814 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 06:55:03.402019 2531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:03.501514 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 2408
I20260621 06:55:03.519671 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 2542
W20260621 06:55:03.519642 2334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.120.193:45313 (error 108)
I20260621 06:55:03.521821 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.520900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:34922)
0621 06:55:03.521519 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:03.521629 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":211,"threads_started":1}
W20260621 06:55:03.522100 2334 catalog_manager.cc:4584] TS 93a0a7df2c69447eb6ba3aa1564b3b02 (127.20.120.193:45313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cbbde6bf5e4d45d48d4781181224a146: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Jun 21 06:55:03 dist-test-slave-gmf2 krb5kdc[2302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:55:03.534353 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.520488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.20.120.194:46049)
0621 06:55:03.520650 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:03.520656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:03.520726 (+ 70us) server_negotiation.cc:407] Connection header received
0621 06:55:03.520921 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:03.520927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:03.521030 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:03.521151 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:03.523020 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.523852 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.525074 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.525431 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.525524 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:03.529496 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:03.529507 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:03.529521 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:03.529551 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:03.531988 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:03.532577 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:03.532583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:03.532586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:03.532704 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:03.532989 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:03.532992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:03.532994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:03.533361 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:03.533572 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:03.533766 (+ 194us) server_negotiation.cc:299] Negotiation successful
0621 06:55:03.534020 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 06:55:03.535018 2665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:03.575676 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.575185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:34936)
0621 06:55:03.575417 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:03.575512 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260621 06:55:03.576725 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.576353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:34942)
0621 06:55:03.576531 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:03.576598 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 06:55:03.621780 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 2542
I20260621 06:55:03.637247 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.636775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:34944)
0621 06:55:03.637015 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:03.637111 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 06:55:03.638412 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.638010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:34948)
0621 06:55:03.638157 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:03.638233 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 06:55:03.638746 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 2675
Jun 21 06:55:03 dist-test-slave-gmf2 krb5kdc[2302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:55:03.654232 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.639477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34852 (local address 127.20.120.195:32887)
0621 06:55:03.639686 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:03.639691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:03.639711 (+ 20us) server_negotiation.cc:407] Connection header received
0621 06:55:03.639866 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:03.639872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:03.639947 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:03.640057 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:03.641405 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.642199 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.643617 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:03.644015 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:03.644162 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:03.648759 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:03.648774 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:03.648790 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:03.648827 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:03.651425 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:03.652090 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:03.652097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:03.652100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:03.652231 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:03.652605 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:03.652611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:03.652615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:03.652946 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:03.653290 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:03.653590 (+ 300us) server_negotiation.cc:299] Negotiation successful
0621 06:55:03.653800 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 06:55:03.654861 2798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:03.719002 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.718451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:34954)
0621 06:55:03.718705 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:03.718828 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260621 06:55:03.720209 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:03.719743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:34958)
0621 06:55:03.719954 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:03.720049 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260621 06:55:03.743013 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 2675
I20260621 06:55:03.759752 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 2315
I20260621 06:55:03.760923 2377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:03.868034 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 2315
I20260621 06:55:03.888415 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34489
--webserver_interface=127.20.120.254
--webserver_port=45761
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:34489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:55:04.200946 2855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:04.201232 2855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:04.201571 2855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:04.208650 2855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:55:04.208801 2855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:04.208920 2855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:04.209031 2855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:55:04.209127 2855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:55:04.218716 2855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:34489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=45761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:04.220081 2855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:04.221451 2855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:04.243749 2861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:04.243692 2864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:04.243713 2862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:04.249660 2855 server_base.cc:1034] running on GCE node
I20260621 06:55:04.250572 2855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:04.252081 2855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:04.253415 2855 hybrid_clock.cc:648] HybridClock initialized: now 1782024904253361 us; error 54 us; skew 500 ppm
I20260621 06:55:04.253855 2855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:04 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:04.261562 2855 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:55:04.263274 2855 webserver.cc:460] Webserver started at http://127.20.120.254:45761/ using document root <none> and password file <none>
I20260621 06:55:04.264053 2855 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:04.264217 2855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:04.269974 2855 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260621 06:55:04.272653 2871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:04.274034 2855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:55:04.274217 2855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "3a787ded34fd42a09ad258378b7144cc"
format_stamp: "Formatted at 2026-06-21 06:55:01 on dist-test-slave-gmf2"
server_key: "3de60edd34f2447a22597f9717604406"
server_key_iv: "1f474d68e94da5d6ef16a67d5620ca4a"
server_key_version: "encryptionkey@0"
I20260621 06:55:04.275276 2855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:04.302615 2855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:04.306622 2855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:04.306893 2855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:04.325589 2855 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:34489
I20260621 06:55:04.325599 2922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:34489 every 8 connection(s)
I20260621 06:55:04.327086 2855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:55:04.329849 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 2855
I20260621 06:55:04.330832 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:45313
--local_ip_for_outbound_sockets=127.20.120.193
--tserver_master_addrs=127.20.120.254:34489
--webserver_port=46275
--webserver_interface=127.20.120.193
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:55:04.335832 2923 sys_catalog.cc:263] Verifying existing consensus state
I20260621 06:55:04.340382 2923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Bootstrap starting.
I20260621 06:55:04.359751 2923 log.cc:824] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:04.371721 2923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 06:55:04.372262 2923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Bootstrap complete.
I20260621 06:55:04.381580 2923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } }
I20260621 06:55:04.382438 2923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a787ded34fd42a09ad258378b7144cc, State: Initialized, Role: FOLLOWER
I20260621 06:55:04.383031 2923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } }
I20260621 06:55:04.383270 2923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:55:04.383375 2923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:55:04.383601 2923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 1 FOLLOWER]: Advancing to term 2
I20260621 06:55:04.387185 2923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } }
I20260621 06:55:04.387670 2923 leader_election.cc:302] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a787ded34fd42a09ad258378b7144cc; no voters:
I20260621 06:55:04.388408 2923 leader_election.cc:288] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 06:55:04.388507 2927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 2 FOLLOWER]: Leader election won for term 2
I20260621 06:55:04.389703 2927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 2 LEADER]: Becoming Leader. State: Replica: 3a787ded34fd42a09ad258378b7144cc, State: Running, Role: LEADER
I20260621 06:55:04.390205 2927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } }
I20260621 06:55:04.390588 2923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:55:04.392642 2928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3a787ded34fd42a09ad258378b7144cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } } }
I20260621 06:55:04.393038 2928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: This master's current role is: LEADER
I20260621 06:55:04.396807 2929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a787ded34fd42a09ad258378b7144cc. Latest consensus state: current_term: 2 leader_uuid: "3a787ded34fd42a09ad258378b7144cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } } }
I20260621 06:55:04.397114 2929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: This master's current role is: LEADER
I20260621 06:55:04.398700 2933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:55:04.406800 2933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:55:04.408371 2933 catalog_manager.cc:1259] Loaded cluster ID: ac1eb1f3191f4f69b5fc38f4b086adba
I20260621 06:55:04.408517 2933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:55:04.411484 2933 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260621 06:55:04.428143 2933 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260621 06:55:04.436692 2933 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:55:04.438282 2933 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Loaded TSK: 0
I20260621 06:55:04.439764 2933 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 06:55:04.668066 2925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:04.668320 2925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:04.668469 2925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:04.675808 2925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:04.675912 2925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:04.676082 2925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:55:04.685868 2925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:45313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=46275
--tserver_master_addrs=127.20.120.254:34489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:04.687184 2925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:04.688486 2925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:04.702814 2950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:04.702840 2951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:04.703711 2953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:04.704556 2925 server_base.cc:1034] running on GCE node
I20260621 06:55:04.705142 2925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:04.706132 2925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:04.707355 2925 hybrid_clock.cc:648] HybridClock initialized: now 1782024904707301 us; error 48 us; skew 500 ppm
I20260621 06:55:04.707661 2925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:04 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:04.710876 2925 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:55:04.712406 2925 webserver.cc:460] Webserver started at http://127.20.120.193:46275/ using document root <none> and password file <none>
I20260621 06:55:04.713104 2925 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:04.713282 2925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:04.718698 2925 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:55:04.721837 2960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:04.723205 2925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:55:04.723372 2925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "93a0a7df2c69447eb6ba3aa1564b3b02"
format_stamp: "Formatted at 2026-06-21 06:55:01 on dist-test-slave-gmf2"
server_key: "cd720c3475dbd21bcd9d7b099e08dac4"
server_key_iv: "e99fcca2a65a69a8334afac08f5beab4"
server_key_version: "encryptionkey@0"
I20260621 06:55:04.724392 2925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:04.735641 2925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:04.739871 2925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:04.740137 2925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:04.741060 2925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:04.743016 2967 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260621 06:55:04.747977 2967 ts_tablet_manager.cc:1852] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260621 06:55:04.748878 2967 ts_tablet_manager.cc:1905] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:04.755632 2967 ts_tablet_manager.cc:1918] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:55:04.755813 2967 log.cc:1196] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/cbbde6bf5e4d45d48d4781181224a146
I20260621 06:55:04.756273 2967 ts_tablet_manager.cc:1939] T cbbde6bf5e4d45d48d4781181224a146 P 93a0a7df2c69447eb6ba3aa1564b3b02: Deleting consensus metadata
I20260621 06:55:04.756699 2925 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260621 06:55:04.756819 2925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.001s
I20260621 06:55:04.756948 2925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:04.757045 2925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:04.789038 2925 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:45313
I20260621 06:55:04.789121 3073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:45313 every 8 connection(s)
I20260621 06:55:04.790337 2925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:55:04.795305 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 2925
I20260621 06:55:04.796036 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:46049
--local_ip_for_outbound_sockets=127.20.120.194
--tserver_master_addrs=127.20.120.254:34489
--webserver_port=37267
--webserver_interface=127.20.120.194
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 06:55:04 dist-test-slave-gmf2 krb5kdc[2302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:04.815918 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:04.793542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:52555)
0621 06:55:04.794319 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:04.794364 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:04.796842 (+ 2478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:04.798177 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:04.798190 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:04.798958 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:55:04.799709 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:04.799735 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:04.801390 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:04.801397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:04.803221 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:04.803232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:04.804629 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:04.804633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:04.804810 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:04.805506 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:55:04.805536 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:55:04.807545 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:55:04.812181 (+ 4636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:04.812187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:04.812197 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:04.812567 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:04.813050 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:04.813053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:04.813055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:04.813238 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:04.814211 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:55:04.814216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:55:04.814484 (+ 268us) client_negotiation.cc:769] Sending connection context
0621 06:55:04.814690 (+ 206us) client_negotiation.cc:240] Negotiation successful
0621 06:55:04.815043 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":345,"threads_started":1}
I20260621 06:55:04.816731 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:04.793249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:52555 (local address 127.20.120.254:34489)
0621 06:55:04.793749 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:04.793761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:04.794493 (+ 732us) server_negotiation.cc:407] Connection header received
0621 06:55:04.797371 (+ 2878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:04.797383 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:04.797695 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:04.798022 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:04.799936 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:04.801236 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:04.803358 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:04.804433 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:04.804634 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:04.807824 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:04.807835 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:04.807855 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:04.807903 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:04.812048 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:04.812714 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:04.812725 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:04.812735 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:04.812942 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:04.813363 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:04.813371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:04.813377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:04.814072 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:04.814296 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:04.814988 (+ 692us) server_negotiation.cc:299] Negotiation successful
0621 06:55:04.815425 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":236,"threads_started":1}
I20260621 06:55:04.819453 3074 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34489
I20260621 06:55:04.819752 3074 heartbeater.cc:464] Registering TS with master...
I20260621 06:55:04.820446 3074 heartbeater.cc:511] Master 127.20.120.254:34489 requested a full tablet report, sending...
I20260621 06:55:04.822557 2888 ts_manager.cc:194] Registered new tserver with Master: 93a0a7df2c69447eb6ba3aa1564b3b02 (127.20.120.193:45313)
I20260621 06:55:04.824146 2888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:52555
W20260621 06:55:05.113304 3078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:05.113646 3078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:05.113811 3078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:05.121122 3078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:05.121266 3078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:05.121425 3078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:55:05.130918 3078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:46049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=37267
--tserver_master_addrs=127.20.120.254:34489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:05.132169 3078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:05.133529 3078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:05.145795 3086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:05.145846 3084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:05.145866 3083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:05.146530 3078 server_base.cc:1034] running on GCE node
I20260621 06:55:05.147051 3078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:05.147886 3078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:05.149093 3078 hybrid_clock.cc:648] HybridClock initialized: now 1782024905149063 us; error 42 us; skew 500 ppm
I20260621 06:55:05.149397 3078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:05 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024905, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:05.152192 3078 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:55:05.153591 3078 webserver.cc:460] Webserver started at http://127.20.120.194:37267/ using document root <none> and password file <none>
I20260621 06:55:05.154309 3078 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:05.154462 3078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:05.159364 3078 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260621 06:55:05.161839 3093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:05.163192 3078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:55:05.163372 3078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "c74b10b35ad74dfa9a6965b863724d97"
format_stamp: "Formatted at 2026-06-21 06:55:02 on dist-test-slave-gmf2"
server_key: "ca8a642ffc594bfbdf69e8166d5a795e"
server_key_iv: "4d036dbde4639689833d32d563880a70"
server_key_version: "encryptionkey@0"
I20260621 06:55:05.164341 3078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:05.185111 3078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:05.188628 3078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:05.188874 3078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:05.189713 3078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:05.190986 3078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:05.191102 3078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:05.191233 3078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:05.191325 3078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:05.223021 3078 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:46049
I20260621 06:55:05.223104 3205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:46049 every 8 connection(s)
I20260621 06:55:05.224390 3078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:55:05.230681 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 3078
I20260621 06:55:05.231309 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:32887
--local_ip_for_outbound_sockets=127.20.120.195
--tserver_master_addrs=127.20.120.254:34489
--webserver_port=43947
--webserver_interface=127.20.120.195
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 06:55:05 dist-test-slave-gmf2 krb5kdc[2302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024905, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:05.245213 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:05.226926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:54181 (local address 127.20.120.254:34489)
0621 06:55:05.227170 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:05.227176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:05.228298 (+ 1122us) server_negotiation.cc:407] Connection header received
0621 06:55:05.230253 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:05.230261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:05.230371 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:05.230492 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:05.231979 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.233126 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.234707 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.235485 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.235616 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:05.239015 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:05.239023 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:05.239030 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:05.239074 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:05.241840 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:05.242722 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:05.242726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:05.242728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:05.242839 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:05.243310 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:05.243317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:05.243320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:05.243712 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:05.243900 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:05.244831 (+ 931us) server_negotiation.cc:299] Negotiation successful
0621 06:55:05.244996 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260621 06:55:05.246613 3208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:05.227471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:54181)
0621 06:55:05.228137 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:05.228181 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:05.229924 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:05.230676 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:05.230684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:05.231216 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:55:05.231805 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:05.231821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.233268 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.233271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:05.234549 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:05.234561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.235692 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.235696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:05.235871 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:05.236541 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:55:05.236566 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:55:05.238826 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:55:05.241992 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:05.242003 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:05.242020 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:05.242617 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:05.242935 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:05.242942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:05.242954 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:05.243179 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:05.243864 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:55:05.243874 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:55:05.244406 (+ 532us) client_negotiation.cc:769] Sending connection context
0621 06:55:05.244722 (+ 316us) client_negotiation.cc:240] Negotiation successful
0621 06:55:05.245251 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":250,"threads_started":1}
I20260621 06:55:05.248111 3206 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34489
I20260621 06:55:05.248416 3206 heartbeater.cc:464] Registering TS with master...
I20260621 06:55:05.249321 3206 heartbeater.cc:511] Master 127.20.120.254:34489 requested a full tablet report, sending...
I20260621 06:55:05.250962 2888 ts_manager.cc:194] Registered new tserver with Master: c74b10b35ad74dfa9a6965b863724d97 (127.20.120.194:46049)
I20260621 06:55:05.251763 2888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:54181
W20260621 06:55:05.553112 3209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:05.553380 3209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:05.553582 3209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:05.560739 3209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:05.560843 3209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:05.561060 3209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:55:05.570719 3209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:32887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=43947
--tserver_master_addrs=127.20.120.254:34489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:05.571956 3209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:05.573318 3209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:05.586611 3214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:05.586470 3215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:05.586714 3217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:05.587687 3209 server_base.cc:1034] running on GCE node
I20260621 06:55:05.588222 3209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:05.589035 3209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:05.590215 3209 hybrid_clock.cc:648] HybridClock initialized: now 1782024905590165 us; error 51 us; skew 500 ppm
I20260621 06:55:05.590590 3209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:05 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024905, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:05.593739 3209 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:55:05.595155 3209 webserver.cc:460] Webserver started at http://127.20.120.195:43947/ using document root <none> and password file <none>
I20260621 06:55:05.595790 3209 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:05.595957 3209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:05.601449 3209 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260621 06:55:05.604772 3224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:05.606138 3209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 06:55:05.606300 3209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "1e4c49a9fe4b419181608e0a84c48682"
format_stamp: "Formatted at 2026-06-21 06:55:02 on dist-test-slave-gmf2"
server_key: "d2022938de9db6ab3b15769ffa2fbdbb"
server_key_iv: "29b72a37cf3b9e100198e4c235b43b38"
server_key_version: "encryptionkey@0"
I20260621 06:55:05.607275 3209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:05.626577 3209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:05.630988 3209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:05.631256 3209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:05.632221 3209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:05.633504 3209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:05.633635 3209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:05.633783 3209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:05.633874 3209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:05.666980 3209 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:32887
I20260621 06:55:05.667045 3336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:32887 every 8 connection(s)
I20260621 06:55:05.668500 3209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:55:05.675937 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 3209
Jun 21 06:55:05 dist-test-slave-gmf2 krb5kdc[2302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024905, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:05.690075 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:05.670977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:44377 (local address 127.20.120.254:34489)
0621 06:55:05.671141 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:05.671144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:05.672180 (+ 1036us) server_negotiation.cc:407] Connection header received
0621 06:55:05.673961 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:05.673968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:05.674050 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:05.674176 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:05.675641 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.676420 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.678649 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.679375 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.679517 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:05.683278 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:05.683282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:05.683286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:05.683316 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:05.686734 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:05.687578 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:05.687584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:05.687589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:05.687744 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:05.688256 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:05.688259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:05.688261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:05.688520 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:05.688692 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:05.689609 (+ 917us) server_negotiation.cc:299] Negotiation successful
0621 06:55:05.689795 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 06:55:05.691360 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:05.671360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:44377)
0621 06:55:05.672012 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:05.672056 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:05.673644 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:05.674432 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:05.674440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:05.674930 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:55:05.675461 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:05.675477 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.676608 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.676612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:05.678512 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:05.678523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.679583 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.679588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:05.679771 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:05.680489 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:55:05.680522 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:55:05.683049 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:55:05.686865 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:05.686873 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:05.686889 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:05.687463 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:05.687884 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:05.687888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:05.687891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:05.688142 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:05.688665 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:55:05.688690 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:55:05.689153 (+ 463us) client_negotiation.cc:769] Sending connection context
0621 06:55:05.689538 (+ 385us) client_negotiation.cc:240] Negotiation successful
0621 06:55:05.690092 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":265,"threads_started":1}
I20260621 06:55:05.692953 3337 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34489
I20260621 06:55:05.693172 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:05.676867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.20.120.254:34489)
0621 06:55:05.677878 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:05.677882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:05.677895 (+ 13us) server_negotiation.cc:407] Connection header received
0621 06:55:05.678008 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:05.678012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:05.678071 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:05.678183 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:05.680197 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.681370 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.682775 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.683448 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.683552 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:05.686840 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:05.686846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:05.686853 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:05.686886 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:05.690280 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:05.690999 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:05.691004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:05.691006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:05.691142 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:05.691606 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:05.691610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:05.691613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:05.691942 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:05.692184 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:05.692588 (+ 404us) server_negotiation.cc:299] Negotiation successful
0621 06:55:05.692765 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":812,"threads_started":1}
I20260621 06:55:05.693271 3337 heartbeater.cc:464] Registering TS with master...
I20260621 06:55:05.694077 3337 heartbeater.cc:511] Master 127.20.120.254:34489 requested a full tablet report, sending...
I20260621 06:55:05.695622 2887 ts_manager.cc:194] Registered new tserver with Master: 1e4c49a9fe4b419181608e0a84c48682 (127.20.120.195:32887)
I20260621 06:55:05.696411 2887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:44377
I20260621 06:55:05.707651 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:55:05.714170 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:05.708491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.20.120.254:34489)
0621 06:55:05.708720 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:05.708729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:05.709050 (+ 321us) server_negotiation.cc:407] Connection header received
0621 06:55:05.709206 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:05.709210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:05.709358 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:05.709482 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:55:05.709872 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.710749 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.712179 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.712598 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.712691 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:05.713140 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:55:05.713615 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:55:05.713708 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:05.713808 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 06:55:05.713867 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 06:55:05.714977 2887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:55:05.722723 2887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:55:05.745503 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:05.734348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:46049 (local address 127.0.0.1:37744)
0621 06:55:05.735400 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:05.735434 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:05.735589 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:05.738516 (+ 2927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:05.738522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:05.738548 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:05.738973 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:05.738989 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.740813 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.740818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:05.742704 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:05.742716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.744442 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.744450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:05.744650 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:05.744720 (+ 70us) client_negotiation.cc:769] Sending connection context
0621 06:55:05.744832 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 06:55:05.745026 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":203,"thread_start_us":702,"threads_started":1}
I20260621 06:55:05.745851 3348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:05.734606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.20.120.194:46049)
0621 06:55:05.737891 (+ 3285us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:05.737898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:05.737954 (+ 56us) server_negotiation.cc:407] Connection header received
0621 06:55:05.738049 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:05.738055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:05.738277 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:05.738518 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:05.739193 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.740655 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.742848 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.744222 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.745069 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:05.745227 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:05.745312 (+ 85us) server_negotiation.cc:299] Negotiation successful
0621 06:55:05.745504 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3138,"thread_start_us":3000,"threads_started":1}
I20260621 06:55:05.746104 3352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:05.736103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47778 (local address 127.20.120.193:45313)
0621 06:55:05.736950 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:05.736957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:05.737376 (+ 419us) server_negotiation.cc:407] Connection header received
0621 06:55:05.737575 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:05.737585 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:05.737827 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:05.738035 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:05.738953 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.740237 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.743169 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.744592 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.744739 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:05.744895 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:05.745596 (+ 701us) server_negotiation.cc:299] Negotiation successful
0621 06:55:05.745752 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":586,"threads_started":1}
I20260621 06:55:05.746449 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:05.735789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:47778)
0621 06:55:05.737267 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:05.737283 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:05.737412 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:05.738279 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:05.738286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:05.738329 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:05.738784 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:05.738795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.740395 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.740400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:05.743019 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:05.743032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.744819 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.744826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:05.745069 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:05.745104 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 06:55:05.745194 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 06:55:05.745323 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1396,"mutex_wait_us":107,"thread_start_us":491,"threads_started":1}
I20260621 06:55:05.746547 3351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:05.734641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:32887 (local address 127.0.0.1:42808)
0621 06:55:05.736313 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:05.736331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:05.736447 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:05.739052 (+ 2605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:05.739058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:05.739083 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:05.739436 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:05.739447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.741204 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.741208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:05.743257 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:05.743265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.744976 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.744982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:05.745237 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:05.745265 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 06:55:05.745331 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 06:55:05.745467 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1533,"mutex_wait_us":51,"thread_start_us":2445,"threads_started":1}
I20260621 06:55:05.746831 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:05.735250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.20.120.195:32887)
0621 06:55:05.738518 (+ 3268us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:05.738525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:05.738555 (+ 30us) server_negotiation.cc:407] Connection header received
0621 06:55:05.738650 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:05.738658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:05.738862 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:05.739059 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:05.739608 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.741088 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.743423 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:05.744750 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:05.745641 (+ 891us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:05.745798 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:05.745867 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 06:55:05.746034 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3115,"thread_start_us":2950,"threads_started":1}
I20260621 06:55:05.748694 3009 tablet_service.cc:1467] Processing CreateTablet for tablet 82e4c2e1a1de4de4a6de6ee2720988fd (DEFAULT_TABLE table=test-table [id=edf0024664d3474bbcd21e076abebb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:55:05.750042 3009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e4c2e1a1de4de4a6de6ee2720988fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:05.749680 3141 tablet_service.cc:1467] Processing CreateTablet for tablet 82e4c2e1a1de4de4a6de6ee2720988fd (DEFAULT_TABLE table=test-table [id=edf0024664d3474bbcd21e076abebb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:55:05.750824 3141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e4c2e1a1de4de4a6de6ee2720988fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:05.750285 3272 tablet_service.cc:1467] Processing CreateTablet for tablet 82e4c2e1a1de4de4a6de6ee2720988fd (DEFAULT_TABLE table=test-table [id=edf0024664d3474bbcd21e076abebb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:55:05.751586 3272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e4c2e1a1de4de4a6de6ee2720988fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:05.760042 3353 tablet_bootstrap.cc:492] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: Bootstrap starting.
I20260621 06:55:05.763176 3354 tablet_bootstrap.cc:492] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97: Bootstrap starting.
I20260621 06:55:05.764340 3353 tablet_bootstrap.cc:654] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:05.765314 3355 tablet_bootstrap.cc:492] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682: Bootstrap starting.
I20260621 06:55:05.766288 3353 log.cc:824] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:05.766510 3354 tablet_bootstrap.cc:654] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:05.767741 3354 log.cc:824] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:05.768532 3355 tablet_bootstrap.cc:654] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:05.769408 3353 tablet_bootstrap.cc:492] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: No bootstrap required, opened a new log
I20260621 06:55:05.769660 3353 ts_tablet_manager.cc:1397] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260621 06:55:05.769706 3355 log.cc:824] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:05.771893 3354 tablet_bootstrap.cc:492] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97: No bootstrap required, opened a new log
I20260621 06:55:05.772157 3354 ts_tablet_manager.cc:1397] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260621 06:55:05.774025 3355 tablet_bootstrap.cc:492] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682: No bootstrap required, opened a new log
I20260621 06:55:05.774322 3355 ts_tablet_manager.cc:1397] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260621 06:55:05.777136 3353 raft_consensus.cc:348] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:05.777485 3353 raft_consensus.cc:374] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:05.777611 3353 raft_consensus.cc:729] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a0a7df2c69447eb6ba3aa1564b3b02, State: Initialized, Role: FOLLOWER
I20260621 06:55:05.778230 3353 consensus_queue.cc:260] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:05.779433 3074 heartbeater.cc:503] Master 127.20.120.254:34489 was elected leader, sending a full tablet report...
I20260621 06:55:05.780018 3353 ts_tablet_manager.cc:1428] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260621 06:55:05.782579 3354 raft_consensus.cc:348] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:05.782975 3354 raft_consensus.cc:374] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:05.783074 3354 raft_consensus.cc:729] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74b10b35ad74dfa9a6965b863724d97, State: Initialized, Role: FOLLOWER
I20260621 06:55:05.783687 3354 consensus_queue.cc:260] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:05.784758 3355 raft_consensus.cc:348] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:05.785192 3355 raft_consensus.cc:374] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:05.785331 3355 raft_consensus.cc:729] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4c49a9fe4b419181608e0a84c48682, State: Initialized, Role: FOLLOWER
I20260621 06:55:05.785411 3354 ts_tablet_manager.cc:1428] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260621 06:55:05.785817 3206 heartbeater.cc:503] Master 127.20.120.254:34489 was elected leader, sending a full tablet report...
I20260621 06:55:05.785997 3355 consensus_queue.cc:260] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:05.787453 3337 heartbeater.cc:503] Master 127.20.120.254:34489 was elected leader, sending a full tablet report...
I20260621 06:55:05.788028 3355 ts_tablet_manager.cc:1428] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260621 06:55:05.792481 3075 tablet.cc:2369] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:55:05.920140 3338 tablet.cc:2369] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:55:05.976342 3207 tablet.cc:2369] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:55:06.014343 3359 raft_consensus.cc:482] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:55:06.014707 3359 raft_consensus.cc:504] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:06.016567 3359 leader_election.cc:288] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c74b10b35ad74dfa9a6965b863724d97 (127.20.120.194:46049), 1e4c49a9fe4b419181608e0a84c48682 (127.20.120.195:32887)
I20260621 06:55:06.025650 3348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.017083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:39083 (local address 127.20.120.194:46049)
0621 06:55:06.017338 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:06.017346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:06.017745 (+ 399us) server_negotiation.cc:407] Connection header received
0621 06:55:06.018124 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:06.018130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:06.018215 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:06.018315 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:06.019066 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.020117 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.023143 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.024394 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.024558 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:06.024634 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:06.025253 (+ 619us) server_negotiation.cc:299] Negotiation successful
0621 06:55:06.025384 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260621 06:55:06.025648 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.017591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:45157 (local address 127.20.120.195:32887)
0621 06:55:06.017793 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:06.017799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:06.017967 (+ 168us) server_negotiation.cc:407] Connection header received
0621 06:55:06.018164 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:06.018168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:06.018234 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:06.018319 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:06.019221 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.020244 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.023141 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.024394 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.024558 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:06.024634 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:06.025250 (+ 616us) server_negotiation.cc:299] Negotiation successful
0621 06:55:06.025384 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 06:55:06.025929 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.016787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:46049 (local address 127.20.120.193:39083)
0621 06:55:06.017604 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:06.017629 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:06.017887 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:06.018361 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:06.018365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:06.018396 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:06.018901 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:06.018920 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.020252 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.020256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:06.022970 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:06.022988 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.024706 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.024711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:06.024999 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:06.025097 (+ 98us) client_negotiation.cc:769] Sending connection context
0621 06:55:06.025204 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 06:55:06.025383 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":243,"thread_start_us":413,"threads_started":1}
I20260621 06:55:06.025950 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.017106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:32887 (local address 127.20.120.193:45157)
0621 06:55:06.017853 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:06.017869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:06.018029 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:06.018366 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:06.018370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:06.018463 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:06.019047 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:06.019061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.020379 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.020384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:06.022977 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:06.022987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.024692 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.024700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:06.024997 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:06.025103 (+ 106us) client_negotiation.cc:769] Sending connection context
0621 06:55:06.025204 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 06:55:06.025383 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":317,"spinlock_wait_cycles":20736,"thread_start_us":420,"threads_started":1}
I20260621 06:55:06.027104 3161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e4c2e1a1de4de4a6de6ee2720988fd" candidate_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74b10b35ad74dfa9a6965b863724d97" is_pre_election: true
I20260621 06:55:06.027748 3161 raft_consensus.cc:2454] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a0a7df2c69447eb6ba3aa1564b3b02 in term 0.
I20260621 06:55:06.027674 3292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e4c2e1a1de4de4a6de6ee2720988fd" candidate_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4c49a9fe4b419181608e0a84c48682" is_pre_election: true
I20260621 06:55:06.028218 3292 raft_consensus.cc:2454] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a0a7df2c69447eb6ba3aa1564b3b02 in term 0.
I20260621 06:55:06.028405 2964 leader_election.cc:302] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a0a7df2c69447eb6ba3aa1564b3b02, c74b10b35ad74dfa9a6965b863724d97; no voters:
I20260621 06:55:06.028859 3359 raft_consensus.cc:2781] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:55:06.029002 3359 raft_consensus.cc:482] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:55:06.029111 3359 raft_consensus.cc:3037] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:06.031296 3359 raft_consensus.cc:504] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:06.031884 3359 leader_election.cc:288] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [CANDIDATE]: Term 1 election: Requested vote from peers c74b10b35ad74dfa9a6965b863724d97 (127.20.120.194:46049), 1e4c49a9fe4b419181608e0a84c48682 (127.20.120.195:32887)
I20260621 06:55:06.032272 3161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e4c2e1a1de4de4a6de6ee2720988fd" candidate_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74b10b35ad74dfa9a6965b863724d97"
I20260621 06:55:06.032428 3292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e4c2e1a1de4de4a6de6ee2720988fd" candidate_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4c49a9fe4b419181608e0a84c48682"
I20260621 06:55:06.032487 3161 raft_consensus.cc:3037] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:06.032634 3292 raft_consensus.cc:3037] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:06.035753 3292 raft_consensus.cc:2454] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a0a7df2c69447eb6ba3aa1564b3b02 in term 1.
I20260621 06:55:06.035753 3161 raft_consensus.cc:2454] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a0a7df2c69447eb6ba3aa1564b3b02 in term 1.
I20260621 06:55:06.036155 2964 leader_election.cc:302] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a0a7df2c69447eb6ba3aa1564b3b02, c74b10b35ad74dfa9a6965b863724d97; no voters:
I20260621 06:55:06.036415 3359 raft_consensus.cc:2781] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:55:06.036805 3359 raft_consensus.cc:686] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 1 LEADER]: Becoming Leader. State: Replica: 93a0a7df2c69447eb6ba3aa1564b3b02, State: Running, Role: LEADER
I20260621 06:55:06.037263 3359 consensus_queue.cc:237] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } } peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } }
I20260621 06:55:06.042001 2888 catalog_manager.cc:5495] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93a0a7df2c69447eb6ba3aa1564b3b02 (127.20.120.193). New cstate: current_term: 1 leader_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93a0a7df2c69447eb6ba3aa1564b3b02" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 45313 } health_report { overall_health: HEALTHY } } }
I20260621 06:55:06.111135 3352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.105418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.20.120.193:45313)
0621 06:55:06.105632 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:06.105638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:06.105656 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:55:06.105879 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:06.105883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:06.106018 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:06.106179 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:55:06.106547 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.107344 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.108833 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.109271 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.109403 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:06.109977 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:55:06.110526 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:55:06.110679 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:06.110783 (+ 104us) server_negotiation.cc:299] Negotiation successful
0621 06:55:06.110864 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 06:55:06.119287 3292 raft_consensus.cc:1261] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682 [term 1 FOLLOWER]: Refusing update from remote peer 93a0a7df2c69447eb6ba3aa1564b3b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:55:06.119364 3161 raft_consensus.cc:1261] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97 [term 1 FOLLOWER]: Refusing update from remote peer 93a0a7df2c69447eb6ba3aa1564b3b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:55:06.120354 3359 consensus_queue.cc:1035] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4c49a9fe4b419181608e0a84c48682" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 32887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:55:06.120692 3364 consensus_queue.cc:1035] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74b10b35ad74dfa9a6965b863724d97" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:55:06.131811 3366 mvcc.cc:205] Tried to move back new op lower bound from 7299174015454781440 to 7299174015133065216. Current Snapshot: MvccSnapshot[applied={T|T < 7299174015454781440}]
I20260621 06:55:06.131920 3367 mvcc.cc:205] Tried to move back new op lower bound from 7299174015454781440 to 7299174015133065216. Current Snapshot: MvccSnapshot[applied={T|T < 7299174015454781440}]
I20260621 06:55:06.149044 2888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:55:06.149317 2888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:55:06.152772 2888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Sending DeleteTablet for 3 replicas of tablet 82e4c2e1a1de4de4a6de6ee2720988fd
I20260621 06:55:06.153867 3272 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e4c2e1a1de4de4a6de6ee2720988fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:55:06 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:42808
I20260621 06:55:06.153964 3141 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e4c2e1a1de4de4a6de6ee2720988fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:55:06 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:37744
I20260621 06:55:06.154021 3009 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e4c2e1a1de4de4a6de6ee2720988fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:55:06 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:47778
I20260621 06:55:06.155370 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 2925
I20260621 06:55:06.166369 3376 tablet_replica.cc:330] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682: stopping tablet replica
I20260621 06:55:06.167205 3376 raft_consensus.cc:2229] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:55:06.168038 3376 raft_consensus.cc:2258] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:55:06.169807 3378 tablet_replica.cc:330] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: stopping tablet replica
I20260621 06:55:06.170259 3378 raft_consensus.cc:2229] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:55:06.170619 3376 ts_tablet_manager.cc:1905] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:06.170902 3377 tablet_replica.cc:330] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97: stopping tablet replica
I20260621 06:55:06.171284 3377 raft_consensus.cc:2229] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:55:06.171720 3378 raft_consensus.cc:2258] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:55:06.171802 3377 raft_consensus.cc:2258] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:55:06.175729 3377 ts_tablet_manager.cc:1905] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:06.176038 3378 ts_tablet_manager.cc:1905] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:06.177932 3376 ts_tablet_manager.cc:1918] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:55:06.178164 3376 log.cc:1196] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/82e4c2e1a1de4de4a6de6ee2720988fd
I20260621 06:55:06.178381 3352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.156554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.20.120.193:45313)
0621 06:55:06.157249 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:06.157258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:06.157279 (+ 21us) server_negotiation.cc:407] Connection header received
0621 06:55:06.157343 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:06.157352 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:06.157459 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:06.157567 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:06.160095 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.161663 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.164420 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.165318 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.165464 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:06.170805 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:06.170819 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:06.170839 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:06.170903 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:06.175032 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:06.175810 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:06.175818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:06.175822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:06.175976 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:06.176435 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:06.176440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:06.176446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:06.176895 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:06.177232 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:06.177551 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 06:55:06.177792 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525}
I20260621 06:55:06.178859 3376 ts_tablet_manager.cc:1939] T 82e4c2e1a1de4de4a6de6ee2720988fd P 1e4c49a9fe4b419181608e0a84c48682: Deleting consensus metadata
I20260621 06:55:06.179365 3069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:06.180825 2872 catalog_manager.cc:4841] TS 1e4c49a9fe4b419181608e0a84c48682 (127.20.120.195:32887): tablet 82e4c2e1a1de4de4a6de6ee2720988fd (table test-table [id=edf0024664d3474bbcd21e076abebb05]) successfully deleted
I20260621 06:55:06.187433 3377 ts_tablet_manager.cc:1918] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:55:06.187554 3377 log.cc:1196] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/82e4c2e1a1de4de4a6de6ee2720988fd
I20260621 06:55:06.188025 3377 ts_tablet_manager.cc:1939] T 82e4c2e1a1de4de4a6de6ee2720988fd P c74b10b35ad74dfa9a6965b863724d97: Deleting consensus metadata
I20260621 06:55:06.189478 2875 catalog_manager.cc:4841] TS c74b10b35ad74dfa9a6965b863724d97 (127.20.120.194:46049): tablet 82e4c2e1a1de4de4a6de6ee2720988fd (table test-table [id=edf0024664d3474bbcd21e076abebb05]) successfully deleted
I20260621 06:55:06.277334 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 2925
W20260621 06:55:06.295224 2875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.120.193:45313 (error 108)
I20260621 06:55:06.295444 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 3078
I20260621 06:55:06.297254 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.296577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:47802)
0621 06:55:06.297010 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:06.297107 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":208,"threads_started":1}
W20260621 06:55:06.297498 2875 catalog_manager.cc:4584] TS 93a0a7df2c69447eb6ba3aa1564b3b02 (127.20.120.193:45313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82e4c2e1a1de4de4a6de6ee2720988fd: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
I20260621 06:55:06.309053 3348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.296287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.20.120.194:46049)
0621 06:55:06.296479 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:06.296483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:06.296501 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:55:06.296691 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:06.296695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:06.296755 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:06.296876 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:06.298400 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.299143 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.300596 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.300990 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.301126 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:06.303696 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:06.303709 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:06.303724 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:06.303754 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:06.306368 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:06.307058 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:06.307064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:06.307066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:06.307209 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:06.307598 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:06.307604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:06.307607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:06.307933 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:06.308237 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:06.308553 (+ 316us) server_negotiation.cc:299] Negotiation successful
0621 06:55:06.308749 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260621 06:55:06.309751 3201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:06.351164 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.350564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:47804)
0621 06:55:06.350850 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:06.350971 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260621 06:55:06.352308 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.351848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:47806)
0621 06:55:06.352038 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:06.352130 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 06:55:06.399152 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 3078
I20260621 06:55:06.412921 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.412460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:47816)
0621 06:55:06.412684 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:06.412770 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260621 06:55:06.413959 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.413556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:47824)
0621 06:55:06.413705 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:06.413780 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260621 06:55:06.417789 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 3209
I20260621 06:55:06.431314 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.418512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.20.120.195:32887)
0621 06:55:06.418704 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:06.418709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:06.418728 (+ 19us) server_negotiation.cc:407] Connection header received
0621 06:55:06.418898 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:06.418904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:06.418988 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:06.419080 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:06.420459 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.421275 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.422613 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:06.423017 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:06.423119 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:06.425651 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:06.425668 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:06.425686 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:06.425727 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:06.428619 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:06.429327 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:06.429333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:06.429335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:06.429463 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:06.429772 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:06.429775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:06.429777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:06.430294 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:06.430544 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:06.430793 (+ 249us) server_negotiation.cc:299] Negotiation successful
0621 06:55:06.430951 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 06:55:06.431978 3332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:06.494975 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.494509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:47830)
0621 06:55:06.494720 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:06.494819 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260621 06:55:06.496025 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:06.495633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:45313 (local address 127.0.0.1:47844)
0621 06:55:06.495815 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:06.495876 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260621 06:55:06.521876 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 3209
I20260621 06:55:06.538228 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 2855
I20260621 06:55:06.539441 2918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:06.655570 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 2855
I20260621 06:55:06.677285 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34489
--webserver_interface=127.20.120.254
--webserver_port=45761
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:34489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:55:06.997205 3385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:06.997493 3385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:06.997582 3385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:07.004777 3385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:55:07.004894 3385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:07.004956 3385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:07.005012 3385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:55:07.005055 3385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:55:07.015375 3385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:34489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=45761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:07.016773 3385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:07.018149 3385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:07.030658 3393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:07.030795 3390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:07.030764 3391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:07.031816 3385 server_base.cc:1034] running on GCE node
I20260621 06:55:07.032648 3385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:07.034235 3385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:07.035468 3385 hybrid_clock.cc:648] HybridClock initialized: now 1782024907035414 us; error 47 us; skew 500 ppm
I20260621 06:55:07.035828 3385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:07 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024907, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:07.039078 3385 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:55:07.040642 3385 webserver.cc:460] Webserver started at http://127.20.120.254:45761/ using document root <none> and password file <none>
I20260621 06:55:07.041352 3385 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:07.041512 3385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:07.047149 3385 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 06:55:07.050181 3400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:07.051537 3385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:55:07.051726 3385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "3a787ded34fd42a09ad258378b7144cc"
format_stamp: "Formatted at 2026-06-21 06:55:01 on dist-test-slave-gmf2"
server_key: "3de60edd34f2447a22597f9717604406"
server_key_iv: "1f474d68e94da5d6ef16a67d5620ca4a"
server_key_version: "encryptionkey@0"
I20260621 06:55:07.052750 3385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:07.075564 3385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:07.080185 3385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:07.080471 3385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:07.099653 3385 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:34489
I20260621 06:55:07.099659 3451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:34489 every 8 connection(s)
I20260621 06:55:07.101110 3385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:55:07.108202 3452 sys_catalog.cc:263] Verifying existing consensus state
I20260621 06:55:07.108345 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 3385
I20260621 06:55:07.109380 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:45313
--local_ip_for_outbound_sockets=127.20.120.193
--tserver_master_addrs=127.20.120.254:34489
--webserver_port=46275
--webserver_interface=127.20.120.193
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:55:07.113409 3452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Bootstrap starting.
I20260621 06:55:07.135694 3452 log.cc:824] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:07.152709 3452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 06:55:07.153327 3452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc: Bootstrap complete.
I20260621 06:55:07.163005 3452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } }
I20260621 06:55:07.163956 3452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a787ded34fd42a09ad258378b7144cc, State: Initialized, Role: FOLLOWER
I20260621 06:55:07.164611 3452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } }
I20260621 06:55:07.164844 3452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:55:07.164971 3452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:55:07.165164 3452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 2 FOLLOWER]: Advancing to term 3
I20260621 06:55:07.169466 3452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } }
I20260621 06:55:07.170061 3452 leader_election.cc:302] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a787ded34fd42a09ad258378b7144cc; no voters:
I20260621 06:55:07.170773 3452 leader_election.cc:288] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 06:55:07.170920 3456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 3 FOLLOWER]: Leader election won for term 3
I20260621 06:55:07.172016 3456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [term 3 LEADER]: Becoming Leader. State: Replica: 3a787ded34fd42a09ad258378b7144cc, State: Running, Role: LEADER
I20260621 06:55:07.172654 3456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } }
I20260621 06:55:07.173089 3452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:55:07.175201 3458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a787ded34fd42a09ad258378b7144cc. Latest consensus state: current_term: 3 leader_uuid: "3a787ded34fd42a09ad258378b7144cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } } }
I20260621 06:55:07.175599 3458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: This master's current role is: LEADER
I20260621 06:55:07.175470 3457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3a787ded34fd42a09ad258378b7144cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a787ded34fd42a09ad258378b7144cc" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34489 } } }
I20260621 06:55:07.175889 3457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a787ded34fd42a09ad258378b7144cc [sys.catalog]: This master's current role is: LEADER
I20260621 06:55:07.183452 3463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:55:07.194671 3463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:55:07.196707 3463 catalog_manager.cc:1259] Loaded cluster ID: ac1eb1f3191f4f69b5fc38f4b086adba
I20260621 06:55:07.196913 3463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260621 06:55:07.199659 3463 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fccd16919ad google::LogMessage::Fail() at ??:0
@ 0x7fccd169488a google::LogMessage::SendToLog() at ??:0
@ 0x7fccd16914de google::LogMessage::Flush() at ??:0
@ 0x7fccd1692a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcce70c4512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcce70c1976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcce7149610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fcce7149439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fccd42dd345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fccd42e091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fccd42e0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fccd42bbeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fccd75956db start_thread at ??:0
@ 0x7fccd052b61f clone at ??:0
W20260621 06:55:07.516531 3454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:07.516840 3454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:07.517038 3454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:07.524858 3454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:07.524963 3454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:07.525178 3454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:55:07.534869 3454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:45313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=46275
--tserver_master_addrs=127.20.120.254:34489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:07.536159 3454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:07.537480 3454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:07.550518 3493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:07.550565 3494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:07.550613 3496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:07.551314 3454 server_base.cc:1034] running on GCE node
I20260621 06:55:07.551862 3454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:07.552778 3454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:07.554061 3454 hybrid_clock.cc:648] HybridClock initialized: now 1782024907553973 us; error 63 us; skew 500 ppm
I20260621 06:55:07.554358 3454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:07 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024907, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:07.557457 3454 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:55:07.558996 3454 webserver.cc:460] Webserver started at http://127.20.120.193:46275/ using document root <none> and password file <none>
I20260621 06:55:07.559698 3454 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:07.559837 3454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:07.565160 3454 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260621 06:55:07.569017 3503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:07.570940 3454 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260621 06:55:07.571126 3454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "93a0a7df2c69447eb6ba3aa1564b3b02"
format_stamp: "Formatted at 2026-06-21 06:55:01 on dist-test-slave-gmf2"
server_key: "cd720c3475dbd21bcd9d7b099e08dac4"
server_key_iv: "e99fcca2a65a69a8334afac08f5beab4"
server_key_version: "encryptionkey@0"
I20260621 06:55:07.572628 3454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:07.590430 3454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:07.594449 3454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:07.594669 3454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:07.595634 3454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:07.597365 3510 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260621 06:55:07.603672 3510 ts_tablet_manager.cc:1852] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260621 06:55:07.604677 3510 ts_tablet_manager.cc:1905] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:07.613224 3510 ts_tablet_manager.cc:1918] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:55:07.613451 3510 log.cc:1196] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/82e4c2e1a1de4de4a6de6ee2720988fd
I20260621 06:55:07.613955 3510 ts_tablet_manager.cc:1939] T 82e4c2e1a1de4de4a6de6ee2720988fd P 93a0a7df2c69447eb6ba3aa1564b3b02: Deleting consensus metadata
I20260621 06:55:07.614338 3454 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260621 06:55:07.614470 3454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260621 06:55:07.614617 3454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:07.614723 3454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:07.649201 3454 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:45313
I20260621 06:55:07.649304 3616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:45313 every 8 connection(s)
I20260621 06:55:07.650636 3454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:55:07.655429 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:07.653565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:55387)
0621 06:55:07.654346 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:07.654595 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":345,"threads_started":1}
W20260621 06:55:07.656080 3617 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:34489 (0 consecutive failures): Network error: Failed to ping master at 127.20.120.254:34489: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
I20260621 06:55:07.656839 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:07.656517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:35405)
0621 06:55:07.656635 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:07.656693 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 06:55:07.657351 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 3454
I20260621 06:55:07.657662 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:07.657343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:54403)
0621 06:55:07.657475 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:07.657552 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 06:55:07.657895 3617 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 06:55:07.658146 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:46049
--local_ip_for_outbound_sockets=127.20.120.194
--tserver_master_addrs=127.20.120.254:34489
--webserver_port=37267
--webserver_interface=127.20.120.194
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:55:07.972085 3620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:07.972389 3620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:07.972607 3620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:07.979736 3620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:07.979847 3620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:07.980019 3620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:55:07.989259 3620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:46049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=37267
--tserver_master_addrs=127.20.120.254:34489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:07.990479 3620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:07.991811 3620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:08.004046 3626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:08.004139 3625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:08.004165 3628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:08.005112 3620 server_base.cc:1034] running on GCE node
I20260621 06:55:08.005637 3620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:08.006517 3620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:08.007716 3620 hybrid_clock.cc:648] HybridClock initialized: now 1782024908007670 us; error 41 us; skew 500 ppm
I20260621 06:55:08.008024 3620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:08 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024908, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:08.011139 3620 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:55:08.012419 3620 webserver.cc:460] Webserver started at http://127.20.120.194:37267/ using document root <none> and password file <none>
I20260621 06:55:08.013018 3620 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:08.013114 3620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:08.018069 3620 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:55:08.020685 3635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:08.021958 3620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:55:08.022140 3620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "c74b10b35ad74dfa9a6965b863724d97"
format_stamp: "Formatted at 2026-06-21 06:55:02 on dist-test-slave-gmf2"
server_key: "ca8a642ffc594bfbdf69e8166d5a795e"
server_key_iv: "4d036dbde4639689833d32d563880a70"
server_key_version: "encryptionkey@0"
I20260621 06:55:08.023077 3620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:08.042619 3620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:08.046896 3620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:08.047241 3620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:08.048264 3620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:08.049659 3620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:08.049777 3620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:08.049942 3620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:08.050041 3620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260621 06:55:08.081820 3620 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:46049
I20260621 06:55:08.081943 3747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:46049 every 8 connection(s)
I20260621 06:55:08.083263 3620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:55:08.087752 3750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:08.086114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:43389)
0621 06:55:08.086752 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:08.086943 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":236,"threads_started":1}
W20260621 06:55:08.088330 3748 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:34489 (0 consecutive failures): Network error: Failed to ping master at 127.20.120.254:34489: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
I20260621 06:55:08.088753 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 3620
I20260621 06:55:08.089128 3750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:08.088819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:51553)
0621 06:55:08.088964 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:08.089025 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 06:55:08.089426 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:32887
--local_ip_for_outbound_sockets=127.20.120.195
--tserver_master_addrs=127.20.120.254:34489
--webserver_port=43947
--webserver_interface=127.20.120.195
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:55:08.090042 3750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:08.089702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:41317)
0621 06:55:08.089828 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:08.089892 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260621 06:55:08.090325 3748 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 06:55:08.400060 3751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:08.400308 3751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:08.400491 3751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:08.407591 3751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:08.407689 3751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:08.407871 3751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:55:08.417349 3751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:32887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=43947
--tserver_master_addrs=127.20.120.254:34489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:08.418661 3751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:08.419932 3751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:08.432703 3756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:08.432792 3759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:08.433815 3751 server_base.cc:1034] running on GCE node
W20260621 06:55:08.434021 3757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:08.434593 3751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:08.435416 3751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:08.436594 3751 hybrid_clock.cc:648] HybridClock initialized: now 1782024908436563 us; error 31 us; skew 500 ppm
I20260621 06:55:08.436866 3751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:08 dist-test-slave-gmf2 krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024908, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:08.439677 3751 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:55:08.441006 3751 webserver.cc:460] Webserver started at http://127.20.120.195:43947/ using document root <none> and password file <none>
I20260621 06:55:08.441663 3751 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:08.441831 3751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:08.446682 3751 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:55:08.449276 3766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:08.450557 3751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:55:08.450724 3751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "1e4c49a9fe4b419181608e0a84c48682"
format_stamp: "Formatted at 2026-06-21 06:55:02 on dist-test-slave-gmf2"
server_key: "d2022938de9db6ab3b15769ffa2fbdbb"
server_key_iv: "29b72a37cf3b9e100198e4c235b43b38"
server_key_version: "encryptionkey@0"
I20260621 06:55:08.451643 3751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:08.484486 3751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:08.488534 3751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:08.488831 3751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:08.489864 3751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:08.491132 3751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:08.491211 3751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:08.491305 3751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:08.491361 3751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:08.525641 3751 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:32887
I20260621 06:55:08.525733 3878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:32887 every 8 connection(s)
I20260621 06:55:08.527233 3751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:55:08.529873 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 3751
I20260621 06:55:08.532227 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:08.530185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:36511)
0621 06:55:08.530948 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:08.531268 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":329,"threads_started":1}
W20260621 06:55:08.532953 3879 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:34489 (0 consecutive failures): Network error: Failed to ping master at 127.20.120.254:34489: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
I20260621 06:55:08.533680 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:08.533402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:58435)
0621 06:55:08.533527 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:08.533587 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 06:55:08.534526 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:08.534236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:57401)
0621 06:55:08.534355 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:08.534423 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 06:55:08.534776 3879 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 06:55:08.659183 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:08.658388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:40141)
0621 06:55:08.658870 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:08.658957 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":212,"threads_started":1}
I20260621 06:55:09.091553 3884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:09.090795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:39771)
0621 06:55:09.091211 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:09.091343 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":205,"threads_started":1}
I20260621 06:55:09.536226 3885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:09.535373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:52853)
0621 06:55:09.535892 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:09.536006 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":277,"threads_started":1}
I20260621 06:55:09.660696 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:09.660010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:51371)
0621 06:55:09.660451 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:09.660546 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":245,"threads_started":1}
I20260621 06:55:10.093127 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:10.092475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:54179)
0621 06:55:10.092890 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:10.092965 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260621 06:55:10.537837 3888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:10.537114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:44493)
0621 06:55:10.537584 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:10.537683 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":246,"threads_started":1}
I20260621 06:55:10.662477 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:10.661638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:39769)
0621 06:55:10.662226 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:10.662309 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":349,"threads_started":1}
I20260621 06:55:11.094797 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:11.094048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:52503)
0621 06:55:11.094520 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:11.094618 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":239,"threads_started":1}
I20260621 06:55:11.539558 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:11.538758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:43675)
0621 06:55:11.539268 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:11.539377 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":287,"threads_started":1}
I20260621 06:55:11.664119 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:11.663353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:33573)
0621 06:55:11.663886 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:11.663963 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":295,"threads_started":1}
I20260621 06:55:12.096431 3893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:12.095672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:49463)
0621 06:55:12.096125 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:12.096211 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":227,"threads_started":1}
I20260621 06:55:12.541205 3894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:12.540461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:45845)
0621 06:55:12.540944 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:12.541027 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":246,"threads_started":1}
I20260621 06:55:12.665884 3895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:12.665146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:50061)
0621 06:55:12.665609 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:12.665721 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":252,"threads_started":1}
I20260621 06:55:13.098155 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:13.097383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:58257)
0621 06:55:13.097845 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:13.097989 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":239,"threads_started":1}
I20260621 06:55:13.543219 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:13.542313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:59437)
0621 06:55:13.542820 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:13.542969 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":282,"threads_started":1}
I20260621 06:55:13.667394 3898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:13.666802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:49015)
0621 06:55:13.667182 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:13.667254 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":201,"threads_started":1}
I20260621 06:55:14.099726 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:14.099046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:53377)
0621 06:55:14.099466 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:14.099569 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":212,"threads_started":1}
I20260621 06:55:14.544925 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:14.544164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:54829)
0621 06:55:14.544647 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:14.544742 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":258,"threads_started":1}
I20260621 06:55:14.669353 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:14.668381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:36741)
0621 06:55:14.668961 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:14.669082 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":303,"threads_started":1}
I20260621 06:55:15.101537 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:15.100691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:47591)
0621 06:55:15.101220 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:15.101305 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":242,"threads_started":1}
I20260621 06:55:15.546715 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:15.545995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:35189)
0621 06:55:15.546447 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:15.546526 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":188,"threads_started":1}
I20260621 06:55:15.670972 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:15.670340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:41451)
0621 06:55:15.670733 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:15.670811 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":208,"threads_started":1}
I20260621 06:55:16.103439 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:16.102558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:51735)
0621 06:55:16.103154 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:16.103258 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":314,"threads_started":1}
I20260621 06:55:16.548624 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:16.547802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:49169)
0621 06:55:16.548358 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:16.548453 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":272,"threads_started":1}
I20260621 06:55:16.672737 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:16.671957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:50775)
0621 06:55:16.672461 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:16.672563 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":261,"threads_started":1}
I20260621 06:55:17.105273 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:17.104481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:60635)
0621 06:55:17.105006 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:17.105101 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":265,"threads_started":1}
I20260621 06:55:17.550477 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:17.549617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:44395)
0621 06:55:17.550203 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:17.550309 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":334,"threads_started":1}
I20260621 06:55:17.674629 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:17.673760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:33533)
0621 06:55:17.674356 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:17.674443 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":367,"threads_started":1}
I20260621 06:55:18.107183 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:18.106468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:40803)
0621 06:55:18.106942 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:18.107025 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":262,"threads_started":1}
I20260621 06:55:18.551980 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:18.551321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:35787)
0621 06:55:18.551748 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:18.551822 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":253,"threads_started":1}
I20260621 06:55:18.676504 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:18.675630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:39047)
0621 06:55:18.676178 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:18.676296 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":331,"threads_started":1}
I20260621 06:55:19.108822 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:19.108127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:33017)
0621 06:55:19.108557 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:19.108642 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":229,"threads_started":1}
I20260621 06:55:19.553498 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:19.552802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:50473)
0621 06:55:19.553238 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:19.553324 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":224,"threads_started":1}
I20260621 06:55:19.678109 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:19.677379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:49027)
0621 06:55:19.677837 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:19.677963 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":227,"threads_started":1}
I20260621 06:55:20.110426 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:20.109666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:53373)
0621 06:55:20.110170 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:20.110264 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":281,"threads_started":1}
I20260621 06:55:20.555105 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:20.554424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:46613)
0621 06:55:20.554871 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:20.554957 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":233,"threads_started":1}
I20260621 06:55:20.679761 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:20.678947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:49817)
0621 06:55:20.679422 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:20.679535 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":219,"threads_started":1}
I20260621 06:55:21.112198 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:21.111441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:51211)
0621 06:55:21.111920 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:21.112008 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":241,"threads_started":1}
I20260621 06:55:21.556887 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:21.556087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:36179)
0621 06:55:21.556615 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:21.556702 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":276,"threads_started":1}
I20260621 06:55:21.681227 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:21.680527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:54515)
0621 06:55:21.680963 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:21.681044 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":208,"threads_started":1}
I20260621 06:55:22.114274 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:22.113302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:43971)
0621 06:55:22.113891 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:22.114035 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":292,"threads_started":1}
I20260621 06:55:22.558701 3924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:22.557835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:44361)
0621 06:55:22.558377 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:22.558472 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":320,"threads_started":1}
I20260621 06:55:22.683374 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:22.682475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.193:36613)
0621 06:55:22.683097 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:22.683192 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":279,"threads_started":1}
I20260621 06:55:23.116117 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:23.115399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.194:42649)
0621 06:55:23.115838 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:23.115934 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":215,"threads_started":1}
I20260621 06:55:23.540674 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 3454
I20260621 06:55:23.557747 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:23.541670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.20.120.193:45313)
0621 06:55:23.542277 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:23.542300 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:23.542365 (+ 65us) server_negotiation.cc:407] Connection header received
0621 06:55:23.544116 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:23.544126 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:23.544553 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:23.544914 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:23.546388 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:23.547552 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:23.548791 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:23.549428 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:23.549556 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:23.552188 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:23.552194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:23.552207 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:23.552246 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:23.555018 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:23.555618 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:23.555625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:23.555630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:23.555795 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:23.556102 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:23.556105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:23.556107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:23.556565 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:23.556796 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:23.557149 (+ 353us) server_negotiation.cc:299] Negotiation successful
0621 06:55:23.557428 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":345,"threads_started":1}
I20260621 06:55:23.559022 3612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:23.560870 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:23.559900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34489 (local address 127.20.120.195:58319)
0621 06:55:23.560441 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:23.560587 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:34489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":257,"threads_started":1}
I20260621 06:55:23.642818 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 3454
I20260621 06:55:23.660327 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 3620
I20260621 06:55:23.678711 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:23.661265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.20.120.194:46049)
0621 06:55:23.661839 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:23.661853 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:23.661888 (+ 35us) server_negotiation.cc:407] Connection header received
0621 06:55:23.663561 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:23.663570 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:23.664006 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:23.664290 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:23.666175 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:23.667710 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:23.669041 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:23.669710 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:23.669832 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:23.672549 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:23.672555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:23.672569 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:23.672600 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:23.675636 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:23.676194 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:23.676199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:23.676205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:23.676348 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:23.676864 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:23.676868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:23.676869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:23.677308 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:23.677533 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:23.678024 (+ 491us) server_negotiation.cc:299] Negotiation successful
0621 06:55:23.678435 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":268,"threads_started":1}
I20260621 06:55:23.680110 3743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:23.763029 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 3620
I20260621 06:55:23.780361 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 3751
I20260621 06:55:23.799077 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:23.781672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.20.120.195:32887)
0621 06:55:23.782396 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:23.782411 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:23.782464 (+ 53us) server_negotiation.cc:407] Connection header received
0621 06:55:23.784442 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:23.784452 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:23.784919 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:23.785233 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:23.786795 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:23.788026 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:23.789311 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:23.790064 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:23.790226 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:23.792787 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:23.792794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:23.792808 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:23.792843 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:23.795894 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:23.796530 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:23.796537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:23.796543 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:23.796745 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:23.797093 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:23.797097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:23.797099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:23.797638 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:23.797891 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:23.798339 (+ 448us) server_negotiation.cc:299] Negotiation successful
0621 06:55:23.798784 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":377,"threads_started":1}
I20260621 06:55:23.800511 3874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:23.883028 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 3751
2026-06-21T06:55:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24516 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:55:23 dist-test-slave-gmf2 krb5kdc[3936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:55:23 dist-test-slave-gmf2 krb5kdc[3936](info): set up 2 sockets
Jun 21 06:55:23 dist-test-slave-gmf2 krb5kdc[3936](info): commencing operation
krb5kdc: starting...
W20260621 06:55:24.978332 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:55:25 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:55:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:55:25.201435 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44667
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:55:25.526500 3949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:25.526779 3949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:25.526866 3949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:25.533797 3949 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 06:55:25.533885 3949 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 06:55:25.533974 3949 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 06:55:25.534046 3949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:55:25.534124 3949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:25.534232 3949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:25.534297 3949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:55:25.534407 3949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:55:25.544070 3949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:25.545331 3949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:25.546725 3949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:25.558749 3954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:25.558710 3957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:25.558770 3955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:25.560057 3949 server_base.cc:1034] running on GCE node
I20260621 06:55:25.560887 3949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:25.562338 3949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:25.563596 3949 hybrid_clock.cc:648] HybridClock initialized: now 1782024925563542 us; error 34 us; skew 500 ppm
I20260621 06:55:25.563917 3949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:25 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024925, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:25.566985 3949 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:55:25.568383 3949 webserver.cc:460] Webserver started at http://127.20.120.254:45933/ using document root <none> and password file <none>
I20260621 06:55:25.569072 3949 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:25.569204 3949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:25.569516 3949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:55:25.572475 3949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "040658089fb548289f671cef93ae862a"
format_stamp: "Formatted at 2026-06-21 06:55:25 on dist-test-slave-gmf2"
server_key: "cdc5d7a978d69a9be7bdbc0ec6d17980"
server_key_iv: "e31f089bfc2c5a19adb76e8e1db481a5"
server_key_version: "encryptionkey@0"
I20260621 06:55:25.573212 3949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "040658089fb548289f671cef93ae862a"
format_stamp: "Formatted at 2026-06-21 06:55:25 on dist-test-slave-gmf2"
server_key: "cdc5d7a978d69a9be7bdbc0ec6d17980"
server_key_iv: "e31f089bfc2c5a19adb76e8e1db481a5"
server_key_version: "encryptionkey@0"
I20260621 06:55:25.578311 3949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260621 06:55:25.581506 3964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:25.582795 3949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:55:25.582963 3949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "040658089fb548289f671cef93ae862a"
format_stamp: "Formatted at 2026-06-21 06:55:25 on dist-test-slave-gmf2"
server_key: "cdc5d7a978d69a9be7bdbc0ec6d17980"
server_key_iv: "e31f089bfc2c5a19adb76e8e1db481a5"
server_key_version: "encryptionkey@0"
I20260621 06:55:25.583169 3949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:25.608399 3949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:25.612299 3949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:25.612569 3949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:25.630474 3949 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:44667
I20260621 06:55:25.630496 4015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:44667 every 8 connection(s)
I20260621 06:55:25.631728 3949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:55:25.635319 4016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:25.641789 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 3949
I20260621 06:55:25.642028 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:55:25.642831 20963 external_mini_cluster.cc:1428] Setting key e7effd8352fcb0b1cd979624ecfb53aa
I20260621 06:55:25.646437 4016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Bootstrap starting.
I20260621 06:55:25.649899 4016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:25.651022 4016 log.cc:824] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:25.654072 4016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: No bootstrap required, opened a new log
Jun 21 06:55:25 dist-test-slave-gmf2 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:25.662729 4016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } }
I20260621 06:55:25.663111 4016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:25.663228 4016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040658089fb548289f671cef93ae862a, State: Initialized, Role: FOLLOWER
I20260621 06:55:25.663950 4016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } }
I20260621 06:55:25.664251 4016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:55:25.664394 4016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:55:25.664573 4016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:25.666975 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:25.644910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.20.120.254:44667)
0621 06:55:25.645699 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:25.645710 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:25.645750 (+ 40us) server_negotiation.cc:407] Connection header received
0621 06:55:25.647565 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:25.647588 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:25.647970 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:25.648303 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:25.650162 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:25.651623 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:25.653454 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:25.654255 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:25.654384 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:25.659267 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:25.659275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:25.659290 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:25.659331 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:25.662505 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:25.663326 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:25.663338 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:25.663347 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:25.663581 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:25.664075 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:25.664079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:25.664081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:25.664843 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:25.665073 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:25.665516 (+ 443us) server_negotiation.cc:299] Negotiation successful
0621 06:55:25.665899 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":378,"threads_started":1}
I20260621 06:55:25.667330 4016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } }
I20260621 06:55:25.667805 4016 leader_election.cc:302] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 040658089fb548289f671cef93ae862a; no voters:
I20260621 06:55:25.668457 4016 leader_election.cc:288] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:55:25.668637 4021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:55:25.669792 4021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 1 LEADER]: Becoming Leader. State: Replica: 040658089fb548289f671cef93ae862a, State: Running, Role: LEADER
I20260621 06:55:25.670478 4021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } }
I20260621 06:55:25.670995 4016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:55:25.673563 4022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "040658089fb548289f671cef93ae862a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } } }
I20260621 06:55:25.673889 4022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: This master's current role is: LEADER
I20260621 06:55:25.675318 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040658089fb548289f671cef93ae862a. Latest consensus state: current_term: 1 leader_uuid: "040658089fb548289f671cef93ae862a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } } }
I20260621 06:55:25.675626 4026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:55:25.675660 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: This master's current role is: LEADER
I20260621 06:55:25.681305 4026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:55:25.691221 4026 catalog_manager.cc:1347] Generated new cluster ID: 82faa5712fe741f9aecf68c38047628a
I20260621 06:55:25.691324 4026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:55:25.709745 4026 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:55:25.711027 4026 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:55:25.718019 4026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Generated new TSK 0
I20260621 06:55:25.719022 4026 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:55:25.781495 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44667
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:55:26.126955 4042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:26.127275 4042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:26.127373 4042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:26.134649 4042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:26.134759 4042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:26.134948 4042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:55:26.144614 4042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:26.146024 4042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:26.147298 4042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:26.160503 4047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:26.160492 4048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:26.160445 4050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:26.161201 4042 server_base.cc:1034] running on GCE node
I20260621 06:55:26.161778 4042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:26.162719 4042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:26.163924 4042 hybrid_clock.cc:648] HybridClock initialized: now 1782024926163855 us; error 62 us; skew 500 ppm
I20260621 06:55:26.164263 4042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:26 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024926, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:26.167680 4042 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:55:26.169248 4042 webserver.cc:460] Webserver started at http://127.20.120.193:42475/ using document root <none> and password file <none>
I20260621 06:55:26.170034 4042 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:26.170181 4042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:26.170493 4042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:55:26.173424 4042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6"
format_stamp: "Formatted at 2026-06-21 06:55:26 on dist-test-slave-gmf2"
server_key: "53338185011ee1b324c780087938c99e"
server_key_iv: "e115044e30904daa1c3090ba8799ff34"
server_key_version: "encryptionkey@0"
I20260621 06:55:26.174355 4042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6"
format_stamp: "Formatted at 2026-06-21 06:55:26 on dist-test-slave-gmf2"
server_key: "53338185011ee1b324c780087938c99e"
server_key_iv: "e115044e30904daa1c3090ba8799ff34"
server_key_version: "encryptionkey@0"
I20260621 06:55:26.180099 4042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:55:26.183732 4057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:26.184986 4042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 06:55:26.185161 4042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6"
format_stamp: "Formatted at 2026-06-21 06:55:26 on dist-test-slave-gmf2"
server_key: "53338185011ee1b324c780087938c99e"
server_key_iv: "e115044e30904daa1c3090ba8799ff34"
server_key_version: "encryptionkey@0"
I20260621 06:55:26.185356 4042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:26.204043 4042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:26.208094 4042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:26.208389 4042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:26.209326 4042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:26.210739 4042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:26.210840 4042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:26.210994 4042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:26.211078 4042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:26.243234 4042 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:38299
I20260621 06:55:26.243273 4169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:38299 every 8 connection(s)
I20260621 06:55:26.244521 4042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:55:26.246704 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 4042
I20260621 06:55:26.246912 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:55:26.247771 20963 external_mini_cluster.cc:1428] Setting key 7919abaf2b34cb990eedaa225312e3b4
Jun 21 06:55:26 dist-test-slave-gmf2 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024926, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:26.264946 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:26.247162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:37751 (local address 127.20.120.254:44667)
0621 06:55:26.247634 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:26.247638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:26.248928 (+ 1290us) server_negotiation.cc:407] Connection header received
0621 06:55:26.251148 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:26.251153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:26.251241 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:26.251361 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:26.252885 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:26.253598 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:26.255229 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:26.255617 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:26.255761 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:26.259433 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:26.259438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:26.259441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:26.259472 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:26.262048 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:26.262730 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:26.262735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:26.262739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:26.262858 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:26.263271 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:26.263276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:26.263279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:26.263540 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:26.263703 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:26.264370 (+ 667us) server_negotiation.cc:299] Negotiation successful
0621 06:55:26.264594 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":264,"threads_started":1}
I20260621 06:55:26.265887 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:26.247887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:37751)
0621 06:55:26.248753 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:26.248814 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:26.250881 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:26.251607 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:26.251615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:26.252141 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:55:26.252704 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:26.252721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:26.253757 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:26.253764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:26.255068 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:26.255081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:26.255858 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:26.255866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:26.256114 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:26.256822 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:55:26.256850 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:55:26.259207 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:55:26.262177 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:26.262186 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:26.262203 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:26.262607 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:26.262965 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:26.262971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:26.262975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:26.263158 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:26.263661 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:55:26.263670 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:55:26.264008 (+ 338us) client_negotiation.cc:769] Sending connection context
0621 06:55:26.264261 (+ 253us) client_negotiation.cc:240] Negotiation successful
0621 06:55:26.264768 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":308,"threads_started":1}
I20260621 06:55:26.267534 4170 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44667
I20260621 06:55:26.267851 4170 heartbeater.cc:464] Registering TS with master...
I20260621 06:55:26.268584 4170 heartbeater.cc:511] Master 127.20.120.254:44667 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
I20260621 06:55:26.270715 3981 ts_manager.cc:194] Registered new tserver with Master: 7b667d43be7a4d55bbf3e7f3a7196cf6 (127.20.120.193:38299)
I20260621 06:55:26.272169 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:37751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:55:26.300668 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44667
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:55:26.621833 4176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:26.622103 4176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:26.622205 4176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:26.629258 4176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:26.629351 4176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:26.629544 4176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:55:26.638887 4176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:26.640101 4176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:26.641475 4176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:26.654042 4181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:26.654146 4182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:26.654071 4184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:26.659318 4176 server_base.cc:1034] running on GCE node
I20260621 06:55:26.659860 4176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:26.660741 4176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:26.661986 4176 hybrid_clock.cc:648] HybridClock initialized: now 1782024926661925 us; error 51 us; skew 500 ppm
I20260621 06:55:26.662302 4176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:26 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024926, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:26.669572 4176 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:55:26.671269 4176 webserver.cc:460] Webserver started at http://127.20.120.194:40181/ using document root <none> and password file <none>
I20260621 06:55:26.672036 4176 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:26.672142 4176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:26.672444 4176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:55:26.675247 4176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "b7546aef7922492fa884eb3f8241eaf6"
format_stamp: "Formatted at 2026-06-21 06:55:26 on dist-test-slave-gmf2"
server_key: "986ada03cc6d4910c3a200b163f277a5"
server_key_iv: "e60e425c5656eab2d52b6e6dc48fac80"
server_key_version: "encryptionkey@0"
I20260621 06:55:26.675900 4176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "b7546aef7922492fa884eb3f8241eaf6"
format_stamp: "Formatted at 2026-06-21 06:55:26 on dist-test-slave-gmf2"
server_key: "986ada03cc6d4910c3a200b163f277a5"
server_key_iv: "e60e425c5656eab2d52b6e6dc48fac80"
server_key_version: "encryptionkey@0"
I20260621 06:55:26.681998 4176 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260621 06:55:26.685117 4191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:26.686416 4176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:55:26.686571 4176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "b7546aef7922492fa884eb3f8241eaf6"
format_stamp: "Formatted at 2026-06-21 06:55:26 on dist-test-slave-gmf2"
server_key: "986ada03cc6d4910c3a200b163f277a5"
server_key_iv: "e60e425c5656eab2d52b6e6dc48fac80"
server_key_version: "encryptionkey@0"
I20260621 06:55:26.686767 4176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:26.701895 4176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:26.706250 4176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:26.706553 4176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:26.707536 4176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:26.708819 4176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:26.708890 4176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:26.709069 4176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:26.709116 4176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:26.741803 4176 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:43303
I20260621 06:55:26.741845 4303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:43303 every 8 connection(s)
I20260621 06:55:26.743100 4176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:55:26.753194 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 4176
I20260621 06:55:26.753410 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:55:26.754037 20963 external_mini_cluster.cc:1428] Setting key b240f029e647633ae9882a9b49d85d8f
Jun 21 06:55:26 dist-test-slave-gmf2 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024926, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:26.762224 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:26.745546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:49791 (local address 127.20.120.254:44667)
0621 06:55:26.745721 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:26.745725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:26.746959 (+ 1234us) server_negotiation.cc:407] Connection header received
0621 06:55:26.748627 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:26.748632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:26.748704 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:26.748791 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:26.750200 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:26.750890 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:26.752306 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:26.752693 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:26.752783 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:26.756681 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:26.756687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:26.756691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:26.756729 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:26.759291 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:26.760008 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:26.760014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:26.760017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:26.760148 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:26.760578 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:26.760587 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:26.760589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:26.760832 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:26.760978 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:26.761632 (+ 654us) server_negotiation.cc:299] Negotiation successful
0621 06:55:26.761844 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 06:55:26.763110 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:26.746077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:49791)
0621 06:55:26.746804 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:26.746846 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:26.748396 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:26.748980 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:26.748987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:26.749469 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:55:26.750058 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:26.750076 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:26.751018 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:26.751021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:26.752189 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:26.752200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:26.752915 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:26.752924 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:26.753214 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:26.753889 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:55:26.753962 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:55:26.756478 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:55:26.759453 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:26.759464 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:26.759478 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:26.759884 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:26.760265 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:26.760270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:26.760274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:26.760458 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:26.760961 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:55:26.760969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:55:26.761281 (+ 312us) client_negotiation.cc:769] Sending connection context
0621 06:55:26.761537 (+ 256us) client_negotiation.cc:240] Negotiation successful
0621 06:55:26.762062 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":257,"threads_started":1}
I20260621 06:55:26.764566 4304 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44667
I20260621 06:55:26.764868 4304 heartbeater.cc:464] Registering TS with master...
I20260621 06:55:26.765561 4304 heartbeater.cc:511] Master 127.20.120.254:44667 requested a full tablet report, sending...
I20260621 06:55:26.767077 3981 ts_manager.cc:194] Registered new tserver with Master: b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303)
I20260621 06:55:26.768311 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:49791
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:55:26.804571 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44667
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:55:27.120080 4309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:27.120352 4309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:27.120445 4309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:27.127435 4309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:27.127527 4309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:27.127629 4309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:55:27.136927 4309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:27.138116 4309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:27.139351 4309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:27.151981 4317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:27.151950 4315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:27.151925 4314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:27.153003 4309 server_base.cc:1034] running on GCE node
I20260621 06:55:27.153551 4309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:27.154461 4309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:27.155644 4309 hybrid_clock.cc:648] HybridClock initialized: now 1782024927155589 us; error 47 us; skew 500 ppm
I20260621 06:55:27.155941 4309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:27 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024927, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:27.158952 4309 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:55:27.160323 4309 webserver.cc:460] Webserver started at http://127.20.120.195:41263/ using document root <none> and password file <none>
I20260621 06:55:27.160967 4309 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:27.161089 4309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:27.161414 4309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:55:27.164212 4309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa"
format_stamp: "Formatted at 2026-06-21 06:55:27 on dist-test-slave-gmf2"
server_key: "7448b426b1cc9e41d11fdad712cb30dd"
server_key_iv: "f78ca84ce78fe69c633ad982f242c236"
server_key_version: "encryptionkey@0"
I20260621 06:55:27.164951 4309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa"
format_stamp: "Formatted at 2026-06-21 06:55:27 on dist-test-slave-gmf2"
server_key: "7448b426b1cc9e41d11fdad712cb30dd"
server_key_iv: "f78ca84ce78fe69c633ad982f242c236"
server_key_version: "encryptionkey@0"
I20260621 06:55:27.169852 4309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260621 06:55:27.173300 4324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:27.174732 4309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:55:27.174882 4309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa"
format_stamp: "Formatted at 2026-06-21 06:55:27 on dist-test-slave-gmf2"
server_key: "7448b426b1cc9e41d11fdad712cb30dd"
server_key_iv: "f78ca84ce78fe69c633ad982f242c236"
server_key_version: "encryptionkey@0"
I20260621 06:55:27.175079 4309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:27.192356 4309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:27.196028 4309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:27.196285 4309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:27.197113 4309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:27.198366 4309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:27.198433 4309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:27.198585 4309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:27.198630 4309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:27.231110 4309 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:35985
I20260621 06:55:27.231156 4436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:35985 every 8 connection(s)
I20260621 06:55:27.232383 4309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:55:27.234272 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 4309
I20260621 06:55:27.234438 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:55:27.234918 20963 external_mini_cluster.cc:1428] Setting key 5e629e0c9be6b46bfb35f0fd38e11af7
Jun 21 06:55:27 dist-test-slave-gmf2 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024927, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:27.252321 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.234923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:58793 (local address 127.20.120.254:44667)
0621 06:55:27.235160 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.235166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.236560 (+ 1394us) server_negotiation.cc:407] Connection header received
0621 06:55:27.238430 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.238434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.238520 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.238601 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:27.239921 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.240669 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.242197 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.242641 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.242759 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.246316 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:27.246323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:27.246327 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:27.246368 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:27.249767 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:27.250487 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:27.250491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:27.250494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:27.250625 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:27.250993 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:27.250996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:27.250998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:27.251244 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:27.251352 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.251937 (+ 585us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.252091 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 06:55:27.253132 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.235522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:58793)
0621 06:55:27.236414 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:27.236459 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:27.238197 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:27.238794 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:27.238801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:27.239300 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:55:27.239792 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.239807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.240808 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.240811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.242080 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.242091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.242870 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.242876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.243077 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.243710 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:55:27.243752 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:55:27.246129 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:55:27.249903 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:27.249911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:27.249943 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:27.250377 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:27.250726 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:27.250729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:27.250731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:27.250897 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:27.251346 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:55:27.251351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:55:27.251627 (+ 276us) client_negotiation.cc:769] Sending connection context
0621 06:55:27.251843 (+ 216us) client_negotiation.cc:240] Negotiation successful
0621 06:55:27.252165 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":372,"threads_started":1}
I20260621 06:55:27.254360 4437 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44667
I20260621 06:55:27.254601 4437 heartbeater.cc:464] Registering TS with master...
I20260621 06:55:27.255275 4437 heartbeater.cc:511] Master 127.20.120.254:44667 requested a full tablet report, sending...
I20260621 06:55:27.256506 3981 ts_manager.cc:194] Registered new tserver with Master: f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985)
I20260621 06:55:27.257215 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:58793
I20260621 06:55:27.262079 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:55:27.275012 4170 heartbeater.cc:503] Master 127.20.120.254:44667 was elected leader, sending a full tablet report...
I20260621 06:55:27.279495 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.266398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37166 (local address 127.20.120.254:44667)
0621 06:55:27.266589 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.266596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.267008 (+ 412us) server_negotiation.cc:407] Connection header received
0621 06:55:27.267126 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.267130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.267183 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.267260 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:27.268642 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.269335 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.270722 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.271189 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.271283 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.273640 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:27.273645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:27.273649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:27.273700 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:27.276749 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:27.277439 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:27.277443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:27.277445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:27.277583 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:27.277953 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:27.277960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:27.277964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:27.278356 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:27.278578 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.278928 (+ 350us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.279124 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 06:55:27.283476 3981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:55:27.290410 3981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:55:27.314121 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.302730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34860 (local address 127.20.120.193:38299)
0621 06:55:27.303696 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.303708 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.304165 (+ 457us) server_negotiation.cc:407] Connection header received
0621 06:55:27.304318 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.304327 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.304597 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.304820 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:27.305870 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.307931 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.311358 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.312741 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.312935 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.313089 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.313604 (+ 515us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.313772 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":722,"threads_started":1}
I20260621 06:55:27.314519 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.302028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.20.120.194:43303)
0621 06:55:27.304152 (+ 2124us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.304161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.304186 (+ 25us) server_negotiation.cc:407] Connection header received
0621 06:55:27.304286 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.304294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.304536 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.305586 (+ 1050us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:27.305879 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.307926 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.311046 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.312564 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.312726 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.312926 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.314007 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.314183 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1933,"thread_start_us":1808,"threads_started":1}
I20260621 06:55:27.314967 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.302147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38299 (local address 127.0.0.1:34860)
0621 06:55:27.304026 (+ 1879us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:27.304043 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:27.304180 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:27.305137 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:27.305162 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:27.305196 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:27.305635 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.305646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.308168 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.308177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.311184 (+ 3007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.311219 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.312962 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.312968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.313376 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.313412 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 06:55:27.314084 (+ 672us) client_negotiation.cc:240] Negotiation successful
0621 06:55:27.314478 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1762,"mutex_wait_us":180,"thread_start_us":1631,"threads_started":1}
I20260621 06:55:27.314988 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.301976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:43303 (local address 127.0.0.1:35440)
0621 06:55:27.302859 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:27.302906 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:27.303100 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:27.304768 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:27.304778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:27.304809 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:27.305325 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.305343 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.308143 (+ 2800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.308151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.310842 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.310859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.313073 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.313079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.313269 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.313350 (+ 81us) client_negotiation.cc:769] Sending connection context
0621 06:55:27.313482 (+ 132us) client_negotiation.cc:240] Negotiation successful
0621 06:55:27.313661 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":25,"thread_start_us":913,"threads_started":1}
I20260621 06:55:27.316730 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.304130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:35985 (local address 127.0.0.1:40306)
0621 06:55:27.309960 (+ 5830us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:27.309977 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:27.310093 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:27.310648 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:27.310655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:27.310680 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:27.311103 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.311116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.312441 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.312447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.314539 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.314552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.315986 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.315992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.316164 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.316195 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 06:55:27.316286 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 06:55:27.316412 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5678,"thread_start_us":4855,"threads_started":1}
I20260621 06:55:27.316828 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.304700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.20.120.195:35985)
0621 06:55:27.308818 (+ 4118us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.308827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.310085 (+ 1258us) server_negotiation.cc:407] Connection header received
0621 06:55:27.310224 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.310231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.310445 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.310629 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:27.311282 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.312292 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.314750 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.315783 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.315992 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.316181 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.316332 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.316491 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3941,"thread_start_us":3809,"threads_started":1}
I20260621 06:55:27.318980 4105 tablet_service.cc:1467] Processing CreateTablet for tablet de3ca0ed53834b0bb68db04c747e3ff6 (DEFAULT_TABLE table=test-table [id=fb1a8631b0a347d7b40396c362b2fffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:55:27.318981 4239 tablet_service.cc:1467] Processing CreateTablet for tablet de3ca0ed53834b0bb68db04c747e3ff6 (DEFAULT_TABLE table=test-table [id=fb1a8631b0a347d7b40396c362b2fffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:55:27.320451 4105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3ca0ed53834b0bb68db04c747e3ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:27.320462 4239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3ca0ed53834b0bb68db04c747e3ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:27.320485 4372 tablet_service.cc:1467] Processing CreateTablet for tablet de3ca0ed53834b0bb68db04c747e3ff6 (DEFAULT_TABLE table=test-table [id=fb1a8631b0a347d7b40396c362b2fffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:55:27.321632 4372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3ca0ed53834b0bb68db04c747e3ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:27.331858 4455 tablet_bootstrap.cc:492] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa: Bootstrap starting.
I20260621 06:55:27.334322 4456 tablet_bootstrap.cc:492] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6: Bootstrap starting.
I20260621 06:55:27.335637 4457 tablet_bootstrap.cc:492] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6: Bootstrap starting.
I20260621 06:55:27.336011 4455 tablet_bootstrap.cc:654] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:27.337036 4455 log.cc:824] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:27.337693 4456 tablet_bootstrap.cc:654] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:27.338891 4456 log.cc:824] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:27.339196 4457 tablet_bootstrap.cc:654] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:27.339798 4455 tablet_bootstrap.cc:492] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa: No bootstrap required, opened a new log
I20260621 06:55:27.340045 4455 ts_tablet_manager.cc:1397] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260621 06:55:27.340308 4457 log.cc:824] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:27.342110 4456 tablet_bootstrap.cc:492] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6: No bootstrap required, opened a new log
I20260621 06:55:27.342375 4456 ts_tablet_manager.cc:1397] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260621 06:55:27.344219 4457 tablet_bootstrap.cc:492] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6: No bootstrap required, opened a new log
I20260621 06:55:27.344452 4457 ts_tablet_manager.cc:1397] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260621 06:55:27.349682 4456 raft_consensus.cc:348] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } }
I20260621 06:55:27.350101 4456 raft_consensus.cc:374] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:27.350200 4456 raft_consensus.cc:729] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7546aef7922492fa884eb3f8241eaf6, State: Initialized, Role: FOLLOWER
I20260621 06:55:27.350090 4455 raft_consensus.cc:348] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } }
I20260621 06:55:27.350478 4455 raft_consensus.cc:374] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:27.350576 4455 raft_consensus.cc:729] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22a2eebf84f4ae2af1b81768b5ec9fa, State: Initialized, Role: FOLLOWER
I20260621 06:55:27.350972 4456 consensus_queue.cc:260] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } }
I20260621 06:55:27.351132 4455 consensus_queue.cc:260] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } }
I20260621 06:55:27.352088 4304 heartbeater.cc:503] Master 127.20.120.254:44667 was elected leader, sending a full tablet report...
I20260621 06:55:27.352715 4456 ts_tablet_manager.cc:1428] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260621 06:55:27.355746 4457 raft_consensus.cc:348] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } }
I20260621 06:55:27.356164 4457 raft_consensus.cc:374] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:27.356261 4457 raft_consensus.cc:729] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b667d43be7a4d55bbf3e7f3a7196cf6, State: Initialized, Role: FOLLOWER
I20260621 06:55:27.356442 4437 heartbeater.cc:503] Master 127.20.120.254:44667 was elected leader, sending a full tablet report...
I20260621 06:55:27.356953 4455 ts_tablet_manager.cc:1428] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20260621 06:55:27.356890 4457 consensus_queue.cc:260] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } }
I20260621 06:55:27.358568 4457 ts_tablet_manager.cc:1428] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260621 06:55:27.404253 4463 raft_consensus.cc:482] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:55:27.404495 4463 raft_consensus.cc:504] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } }
I20260621 06:55:27.406069 4463 leader_election.cc:288] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303), f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985)
I20260621 06:55:27.408571 4462 raft_consensus.cc:482] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:55:27.408881 4462 raft_consensus.cc:504] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } }
I20260621 06:55:27.410900 4462 leader_election.cc:288] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303), 7b667d43be7a4d55bbf3e7f3a7196cf6 (127.20.120.193:38299)
I20260621 06:55:27.415902 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.406357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:41609 (local address 127.20.120.194:43303)
0621 06:55:27.406557 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.406564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.407399 (+ 835us) server_negotiation.cc:407] Connection header received
0621 06:55:27.407534 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.407540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.407617 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.407700 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:27.408396 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.409509 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.412221 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.413903 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.414075 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.414189 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.415393 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.415578 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 06:55:27.415917 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.406481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:43303 (local address 127.20.120.193:41609)
0621 06:55:27.407288 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:27.407307 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:27.407415 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:27.407743 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:27.407747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:27.407778 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:27.408245 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.408255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.409658 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.409665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.412020 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.412037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.414420 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.414429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.414769 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.414821 (+ 52us) client_negotiation.cc:769] Sending connection context
0621 06:55:27.415358 (+ 537us) client_negotiation.cc:240] Negotiation successful
0621 06:55:27.415558 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":11,"thread_start_us":478,"threads_started":1}
I20260621 06:55:27.416646 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.406521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:53851 (local address 127.20.120.195:35985)
0621 06:55:27.406733 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.406740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.407554 (+ 814us) server_negotiation.cc:407] Connection header received
0621 06:55:27.407721 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.407727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.407799 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.407922 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:27.408629 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.409539 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.412396 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.414135 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.414296 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.414401 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.416213 (+ 1812us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.416357 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 06:55:27.417371 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.406511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:35985 (local address 127.20.120.193:53851)
0621 06:55:27.407440 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:27.407455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:27.407576 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:27.407944 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:27.407947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:27.407965 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:27.408446 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.408462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.409726 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.409730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.412148 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.412214 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.414650 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.414659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.415171 (+ 512us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.415217 (+ 46us) client_negotiation.cc:769] Sending connection context
0621 06:55:27.416807 (+ 1590us) client_negotiation.cc:240] Negotiation successful
0621 06:55:27.416942 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":238,"thread_start_us":500,"threads_started":1}
I20260621 06:55:27.417091 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3ca0ed53834b0bb68db04c747e3ff6" candidate_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7546aef7922492fa884eb3f8241eaf6" is_pre_election: true
I20260621 06:55:27.417716 4259 raft_consensus.cc:2454] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b667d43be7a4d55bbf3e7f3a7196cf6 in term 0.
I20260621 06:55:27.418702 4059 leader_election.cc:302] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b667d43be7a4d55bbf3e7f3a7196cf6, b7546aef7922492fa884eb3f8241eaf6; no voters:
I20260621 06:55:27.419143 4463 raft_consensus.cc:2781] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:55:27.419273 4463 raft_consensus.cc:482] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:55:27.419070 4392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3ca0ed53834b0bb68db04c747e3ff6" candidate_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" is_pre_election: true
I20260621 06:55:27.419359 4463 raft_consensus.cc:3037] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:27.419536 4392 raft_consensus.cc:2454] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b667d43be7a4d55bbf3e7f3a7196cf6 in term 0.
I20260621 06:55:27.422264 4463 raft_consensus.cc:504] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } }
I20260621 06:55:27.423023 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.411077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:43303 (local address 127.20.120.195:51775)
0621 06:55:27.411226 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:27.411254 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:27.411367 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:27.416187 (+ 4820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:27.416191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:27.416219 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:27.416436 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.416449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.418398 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.418403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.420570 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.420581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.422190 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.422196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.422408 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.422446 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 06:55:27.422544 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 06:55:27.422685 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":26}
I20260621 06:55:27.423168 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.411824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:44287 (local address 127.20.120.193:38299)
0621 06:55:27.411988 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.411995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.415286 (+ 3291us) server_negotiation.cc:407] Connection header received
0621 06:55:27.416100 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.416106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.416180 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.416257 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:27.417061 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.418193 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.420900 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.422206 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.422348 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.422421 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.422818 (+ 397us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.422932 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 06:55:27.423313 4463 leader_election.cc:288] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 1 election: Requested vote from peers b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303), f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985)
I20260621 06:55:27.423434 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3ca0ed53834b0bb68db04c747e3ff6" candidate_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7546aef7922492fa884eb3f8241eaf6"
I20260621 06:55:27.423635 4259 raft_consensus.cc:3037] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:27.423782 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.411497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:51775 (local address 127.20.120.194:43303)
0621 06:55:27.415857 (+ 4360us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.415862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.415885 (+ 23us) server_negotiation.cc:407] Connection header received
0621 06:55:27.415947 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.415953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.416024 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.416589 (+ 565us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:27.416915 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.418219 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.420738 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.422035 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.422226 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.422313 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.423239 (+ 926us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.423375 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4240,"thread_start_us":4030,"threads_started":1}
I20260621 06:55:27.424124 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.411488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38299 (local address 127.20.120.195:44287)
0621 06:55:27.415184 (+ 3696us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:27.415198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:27.415334 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:27.416396 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:27.416400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:27.416424 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:27.416863 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.416876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.418377 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.418385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.420737 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:27.420751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.422494 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.422503 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:27.422649 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.422672 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 06:55:27.422772 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 06:55:27.422904 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3567,"mutex_wait_us":24,"thread_start_us":3376,"threads_started":1}
I20260621 06:55:27.424274 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3ca0ed53834b0bb68db04c747e3ff6" candidate_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7546aef7922492fa884eb3f8241eaf6" is_pre_election: true
I20260621 06:55:27.424537 4392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3ca0ed53834b0bb68db04c747e3ff6" candidate_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa"
I20260621 06:55:27.424750 4392 raft_consensus.cc:3037] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:27.426502 4259 raft_consensus.cc:2454] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b667d43be7a4d55bbf3e7f3a7196cf6 in term 1.
I20260621 06:55:27.427019 4059 leader_election.cc:302] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b667d43be7a4d55bbf3e7f3a7196cf6, b7546aef7922492fa884eb3f8241eaf6; no voters:
I20260621 06:55:27.427289 4392 raft_consensus.cc:2454] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b667d43be7a4d55bbf3e7f3a7196cf6 in term 1.
I20260621 06:55:27.427320 4125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3ca0ed53834b0bb68db04c747e3ff6" candidate_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" is_pre_election: true
I20260621 06:55:27.427760 4125 raft_consensus.cc:2379] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f22a2eebf84f4ae2af1b81768b5ec9fa in current term 1: Already voted for candidate 7b667d43be7a4d55bbf3e7f3a7196cf6 in this term.
I20260621 06:55:27.428079 4463 raft_consensus.cc:2781] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:55:27.428364 4328 leader_election.cc:302] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f22a2eebf84f4ae2af1b81768b5ec9fa; no voters: 7b667d43be7a4d55bbf3e7f3a7196cf6, b7546aef7922492fa884eb3f8241eaf6
I20260621 06:55:27.428679 4462 raft_consensus.cc:2726] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 06:55:27.428661 4463 raft_consensus.cc:686] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 LEADER]: Becoming Leader. State: Replica: 7b667d43be7a4d55bbf3e7f3a7196cf6, State: Running, Role: LEADER
I20260621 06:55:27.429196 4463 consensus_queue.cc:237] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } }
I20260621 06:55:27.433673 3980 catalog_manager.cc:5495] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b667d43be7a4d55bbf3e7f3a7196cf6 (127.20.120.193). New cstate: current_term: 1 leader_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } health_report { overall_health: UNKNOWN } } }
I20260621 06:55:27.469300 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.464118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.20.120.193:38299)
0621 06:55:27.464287 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.464291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.464440 (+ 149us) server_negotiation.cc:407] Connection header received
0621 06:55:27.464679 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.464683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.464765 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.464882 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:55:27.465222 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.466031 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.467430 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.467841 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.467929 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.468343 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:55:27.468797 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:55:27.468898 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.469007 (+ 109us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.469067 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 06:55:27.476847 4392 raft_consensus.cc:1261] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [term 1 FOLLOWER]: Refusing update from remote peer 7b667d43be7a4d55bbf3e7f3a7196cf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:55:27.476859 4259 raft_consensus.cc:1261] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6 [term 1 FOLLOWER]: Refusing update from remote peer 7b667d43be7a4d55bbf3e7f3a7196cf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:55:27.477773 4463 consensus_queue.cc:1035] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:55:27.478102 4468 consensus_queue.cc:1035] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 06:55:27.483971 4438 tablet.cc:2369] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:55:27.489718 4471 mvcc.cc:205] Tried to move back new op lower bound from 7299174102936854528 to 7299174102754525184. Current Snapshot: MvccSnapshot[applied={T|T < 7299174102936854528}]
I20260621 06:55:27.505678 3981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:55:27.505991 3981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:55:27.509197 3981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Sending DeleteTablet for 3 replicas of tablet de3ca0ed53834b0bb68db04c747e3ff6
I20260621 06:55:27.510360 4239 tablet_service.cc:1514] Processing DeleteTablet for tablet de3ca0ed53834b0bb68db04c747e3ff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:55:27 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:35440
I20260621 06:55:27.510429 4105 tablet_service.cc:1514] Processing DeleteTablet for tablet de3ca0ed53834b0bb68db04c747e3ff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:55:27 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:34860
I20260621 06:55:27.510437 4372 tablet_service.cc:1514] Processing DeleteTablet for tablet de3ca0ed53834b0bb68db04c747e3ff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:55:27 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:40306
I20260621 06:55:27.511437 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 4042
I20260621 06:55:27.522742 4482 tablet_replica.cc:330] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6: stopping tablet replica
I20260621 06:55:27.522751 4481 tablet_replica.cc:330] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6: stopping tablet replica
I20260621 06:55:27.523213 4481 raft_consensus.cc:2229] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:55:27.523213 4482 raft_consensus.cc:2229] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:55:27.523722 4482 raft_consensus.cc:2258] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:55:27.523931 4481 raft_consensus.cc:2258] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:55:27.525955 4482 ts_tablet_manager.cc:1905] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:27.526000 4481 ts_tablet_manager.cc:1905] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:27.532925 4482 ts_tablet_manager.cc:1918] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:55:27.532994 4481 ts_tablet_manager.cc:1918] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:55:27.533079 4482 log.cc:1196] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/de3ca0ed53834b0bb68db04c747e3ff6
I20260621 06:55:27.533142 4481 log.cc:1196] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/de3ca0ed53834b0bb68db04c747e3ff6
I20260621 06:55:27.533627 4482 ts_tablet_manager.cc:1939] T de3ca0ed53834b0bb68db04c747e3ff6 P b7546aef7922492fa884eb3f8241eaf6: Deleting consensus metadata
I20260621 06:55:27.533629 4481 ts_tablet_manager.cc:1939] T de3ca0ed53834b0bb68db04c747e3ff6 P 7b667d43be7a4d55bbf3e7f3a7196cf6: Deleting consensus metadata
I20260621 06:55:27.534396 4483 tablet_replica.cc:330] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa: stopping tablet replica
I20260621 06:55:27.534832 4483 raft_consensus.cc:2229] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:55:27.535362 4483 raft_consensus.cc:2258] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:55:27.535543 3966 catalog_manager.cc:4841] TS b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303): tablet de3ca0ed53834b0bb68db04c747e3ff6 (table test-table [id=fb1a8631b0a347d7b40396c362b2fffa]) successfully deleted
I20260621 06:55:27.537319 3968 catalog_manager.cc:4841] TS 7b667d43be7a4d55bbf3e7f3a7196cf6 (127.20.120.193:38299): tablet de3ca0ed53834b0bb68db04c747e3ff6 (table test-table [id=fb1a8631b0a347d7b40396c362b2fffa]) successfully deleted
I20260621 06:55:27.537595 4483 ts_tablet_manager.cc:1905] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:27.544171 4483 ts_tablet_manager.cc:1918] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:55:27.544343 4483 log.cc:1196] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/de3ca0ed53834b0bb68db04c747e3ff6
I20260621 06:55:27.544857 4483 ts_tablet_manager.cc:1939] T de3ca0ed53834b0bb68db04c747e3ff6 P f22a2eebf84f4ae2af1b81768b5ec9fa: Deleting consensus metadata
Jun 21 06:55:27 dist-test-slave-gmf2 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:55:27.546432 3968 catalog_manager.cc:4841] TS f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985): tablet de3ca0ed53834b0bb68db04c747e3ff6 (table test-table [id=fb1a8631b0a347d7b40396c362b2fffa]) successfully deleted
I20260621 06:55:27.552893 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.513642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.20.120.193:38299)
0621 06:55:27.513833 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.513839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.534747 (+ 20908us) server_negotiation.cc:407] Connection header received
0621 06:55:27.535261 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.535266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.535350 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.535419 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:27.537504 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.538655 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.540791 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.541468 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.541619 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.547006 (+ 5387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:27.547035 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:27.547053 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:27.547112 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:27.550171 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:27.551014 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:27.551021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:27.551023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:27.551139 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:27.551543 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:27.551546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:27.551548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:27.551916 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:27.552147 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.552475 (+ 328us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.552641 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 06:55:27.553784 4165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:27.652974 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 4042
I20260621 06:55:27.667898 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 4176
Jun 21 06:55:27 dist-test-slave-gmf2 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:55:27.683939 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.668603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.20.120.194:43303)
0621 06:55:27.668776 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.668780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.668797 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:55:27.668941 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.668946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.669006 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.669103 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:27.670416 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.671147 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.672669 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.673071 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.673213 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.677528 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:27.677540 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:27.677555 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:27.677585 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:27.680791 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:27.681676 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:27.681686 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:27.681689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:27.681849 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:27.682342 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:27.682350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:27.682354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:27.682856 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:27.683184 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.683399 (+ 215us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.683607 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 06:55:27.684545 4299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:27.771193 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 4176
I20260621 06:55:27.790192 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 4309
Jun 21 06:55:27 dist-test-slave-gmf2 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:55:27.804407 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:27.790978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.20.120.195:35985)
0621 06:55:27.791156 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:27.791161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:27.791178 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:55:27.791252 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:27.791256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:27.791317 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:27.791392 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:27.792614 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.793376 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.794741 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:27.795116 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:27.795201 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:27.799264 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:27.799278 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:27.799294 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:27.799329 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:27.802096 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:27.802646 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:27.802664 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:27.802666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:27.802806 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:27.803117 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:27.803120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:27.803122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:27.803436 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:27.803640 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:27.803870 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 06:55:27.804052 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:55:27.805079 4432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:27.896279 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 4309
I20260621 06:55:27.912730 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 3949
I20260621 06:55:27.913867 4011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:28.020907 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 3949
I20260621 06:55:48.042302 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44667
--webserver_interface=127.20.120.254
--webserver_port=45933
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:55:48.376528 4489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:48.376811 4489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:48.376899 4489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:48.384084 4489 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 06:55:48.384184 4489 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 06:55:48.384264 4489 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 06:55:48.384353 4489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:55:48.384452 4489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:48.384573 4489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:48.384653 4489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:55:48.384735 4489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:55:48.394552 4489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=45933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:48.395851 4489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:48.397234 4489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:48.420042 4496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:48.420024 4498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:48.420068 4495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:48.420800 4489 server_base.cc:1034] running on GCE node
I20260621 06:55:48.421764 4489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:48.423556 4489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:48.424813 4489 hybrid_clock.cc:648] HybridClock initialized: now 1782024948424741 us; error 46 us; skew 500 ppm
I20260621 06:55:48.425235 4489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:48 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024948, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:48.433708 4489 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:55:48.435551 4489 webserver.cc:460] Webserver started at http://127.20.120.254:45933/ using document root <none> and password file <none>
I20260621 06:55:48.436367 4489 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:48.436517 4489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:48.442672 4489 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 06:55:48.445657 4505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:48.447101 4489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 06:55:48.447299 4489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "040658089fb548289f671cef93ae862a"
format_stamp: "Formatted at 2026-06-21 06:55:25 on dist-test-slave-gmf2"
server_key: "cdc5d7a978d69a9be7bdbc0ec6d17980"
server_key_iv: "e31f089bfc2c5a19adb76e8e1db481a5"
server_key_version: "encryptionkey@0"
I20260621 06:55:48.448343 4489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:48.462510 4489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:48.467327 4489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:48.467628 4489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:48.486811 4489 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:44667
I20260621 06:55:48.486842 4556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:44667 every 8 connection(s)
I20260621 06:55:48.488417 4489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:55:48.495584 4557 sys_catalog.cc:263] Verifying existing consensus state
I20260621 06:55:48.496260 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 4489
I20260621 06:55:48.497097 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:38299
--local_ip_for_outbound_sockets=127.20.120.193
--tserver_master_addrs=127.20.120.254:44667
--webserver_port=42475
--webserver_interface=127.20.120.193
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:55:48.499125 4557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Bootstrap starting.
I20260621 06:55:48.519007 4557 log.cc:824] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:48.531134 4557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 06:55:48.531666 4557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Bootstrap complete.
I20260621 06:55:48.540637 4557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } }
I20260621 06:55:48.541487 4557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040658089fb548289f671cef93ae862a, State: Initialized, Role: FOLLOWER
I20260621 06:55:48.542160 4557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } }
I20260621 06:55:48.542411 4557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:55:48.542560 4557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:55:48.542887 4557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 1 FOLLOWER]: Advancing to term 2
I20260621 06:55:48.546799 4557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } }
I20260621 06:55:48.547333 4557 leader_election.cc:302] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 040658089fb548289f671cef93ae862a; no voters:
I20260621 06:55:48.548002 4557 leader_election.cc:288] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 06:55:48.548151 4561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 2 FOLLOWER]: Leader election won for term 2
I20260621 06:55:48.549285 4561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 2 LEADER]: Becoming Leader. State: Replica: 040658089fb548289f671cef93ae862a, State: Running, Role: LEADER
I20260621 06:55:48.549860 4561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } }
I20260621 06:55:48.550414 4557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:55:48.552328 4563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040658089fb548289f671cef93ae862a. Latest consensus state: current_term: 2 leader_uuid: "040658089fb548289f671cef93ae862a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } } }
I20260621 06:55:48.552677 4563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: This master's current role is: LEADER
I20260621 06:55:48.553325 4562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "040658089fb548289f671cef93ae862a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } } }
I20260621 06:55:48.553610 4562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: This master's current role is: LEADER
I20260621 06:55:48.563156 4566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:55:48.571612 4566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:55:48.572974 4566 catalog_manager.cc:1259] Loaded cluster ID: 82faa5712fe741f9aecf68c38047628a
I20260621 06:55:48.573118 4566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:55:48.577634 4566 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:55:48.579203 4566 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Loaded TSK: 0
I20260621 06:55:48.607913 4566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Generated new TSK 1
I20260621 06:55:48.608286 4566 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 06:55:48.841602 4559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:48.841943 4559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:48.842115 4559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:48.849774 4559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:48.849956 4559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:48.850142 4559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:55:48.860596 4559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:38299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=42475
--tserver_master_addrs=127.20.120.254:44667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:48.861886 4559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:48.863221 4559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:48.878671 4585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:48.878685 4584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:48.878767 4587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:48.880178 4559 server_base.cc:1034] running on GCE node
I20260621 06:55:48.880745 4559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:48.881657 4559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:48.882874 4559 hybrid_clock.cc:648] HybridClock initialized: now 1782024948882822 us; error 50 us; skew 500 ppm
I20260621 06:55:48.883175 4559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:48 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024948, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:48.886327 4559 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:55:48.887806 4559 webserver.cc:460] Webserver started at http://127.20.120.193:42475/ using document root <none> and password file <none>
I20260621 06:55:48.888466 4559 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:48.888600 4559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:48.893778 4559 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:55:48.896368 4594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:48.897732 4559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:55:48.897884 4559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6"
format_stamp: "Formatted at 2026-06-21 06:55:26 on dist-test-slave-gmf2"
server_key: "53338185011ee1b324c780087938c99e"
server_key_iv: "e115044e30904daa1c3090ba8799ff34"
server_key_version: "encryptionkey@0"
I20260621 06:55:48.898895 4559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:48.915982 4559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:48.920251 4559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:48.920519 4559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:48.921542 4559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:48.922997 4559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:48.923090 4559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:48.923203 4559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:48.923306 4559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:48.973773 4559 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:38299
I20260621 06:55:48.973873 4706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:38299 every 8 connection(s)
I20260621 06:55:48.975198 4559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:55:48.980780 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 4559
I20260621 06:55:48.981557 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:43303
--local_ip_for_outbound_sockets=127.20.120.194
--tserver_master_addrs=127.20.120.254:44667
--webserver_port=40181
--webserver_interface=127.20.120.194
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 06:55:48 dist-test-slave-gmf2 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024948, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:49.000205 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:48.978185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:49377)
0621 06:55:48.978934 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:48.978974 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:48.981030 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:48.982519 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:48.982533 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:48.983316 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:55:48.984045 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:48.984070 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:48.985621 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:48.985629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:48.987764 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:48.987780 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:48.988694 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:48.988703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:48.989024 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:48.989734 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:55:48.989755 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:55:48.992154 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:55:48.996484 (+ 4330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:48.996490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:48.996499 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:48.996899 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:48.997371 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:48.997374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:48.997376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:48.997542 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:48.998457 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:55:48.998462 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:55:48.998717 (+ 255us) client_negotiation.cc:769] Sending connection context
0621 06:55:48.998910 (+ 193us) client_negotiation.cc:240] Negotiation successful
0621 06:55:48.999242 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":309,"threads_started":1}
I20260621 06:55:49.000820 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:48.977859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:49377 (local address 127.20.120.254:44667)
0621 06:55:48.978437 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:48.978448 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:48.979103 (+ 655us) server_negotiation.cc:407] Connection header received
0621 06:55:48.981580 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:48.981591 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:48.982035 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:48.982309 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:48.984233 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:48.985447 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:48.987892 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:48.988487 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:48.988600 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:48.992456 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:48.992467 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:48.992487 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:48.992535 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:48.996339 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:48.997054 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:48.997064 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:48.997074 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:48.997266 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:48.997665 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:48.997672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:48.997677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:48.998330 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:48.998498 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:48.999154 (+ 656us) server_negotiation.cc:299] Negotiation successful
0621 06:55:48.999584 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":241,"threads_started":1}
I20260621 06:55:49.003881 4707 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44667
I20260621 06:55:49.004165 4707 heartbeater.cc:464] Registering TS with master...
I20260621 06:55:49.004848 4707 heartbeater.cc:511] Master 127.20.120.254:44667 requested a full tablet report, sending...
I20260621 06:55:49.007342 4522 ts_manager.cc:194] Registered new tserver with Master: 7b667d43be7a4d55bbf3e7f3a7196cf6 (127.20.120.193:38299)
I20260621 06:55:49.009562 4522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:49377
W20260621 06:55:49.309621 4711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:49.309877 4711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:49.310014 4711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:49.317118 4711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:49.317214 4711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:49.317358 4711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:55:49.326673 4711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:43303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=40181
--tserver_master_addrs=127.20.120.254:44667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:49.327808 4711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:49.329018 4711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:49.342427 4717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:49.342458 4716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:49.342376 4719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:49.347554 4711 server_base.cc:1034] running on GCE node
I20260621 06:55:49.348121 4711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:49.348964 4711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:49.350153 4711 hybrid_clock.cc:648] HybridClock initialized: now 1782024949350118 us; error 33 us; skew 500 ppm
I20260621 06:55:49.350459 4711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:49 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024949, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:49.353780 4711 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:55:49.355129 4711 webserver.cc:460] Webserver started at http://127.20.120.194:40181/ using document root <none> and password file <none>
I20260621 06:55:49.355754 4711 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:49.355858 4711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:49.360764 4711 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260621 06:55:49.363333 4726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:49.364678 4711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 06:55:49.364892 4711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "b7546aef7922492fa884eb3f8241eaf6"
format_stamp: "Formatted at 2026-06-21 06:55:26 on dist-test-slave-gmf2"
server_key: "986ada03cc6d4910c3a200b163f277a5"
server_key_iv: "e60e425c5656eab2d52b6e6dc48fac80"
server_key_version: "encryptionkey@0"
I20260621 06:55:49.365902 4711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:49.389158 4711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:49.392915 4711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:49.393177 4711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:49.394146 4711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:49.395493 4711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:49.395613 4711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:49.395746 4711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:49.395831 4711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:49.428627 4711 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:43303
I20260621 06:55:49.428710 4838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:43303 every 8 connection(s)
I20260621 06:55:49.430090 4711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:55:49.438653 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 4711
I20260621 06:55:49.439563 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:35985
--local_ip_for_outbound_sockets=127.20.120.195
--tserver_master_addrs=127.20.120.254:44667
--webserver_port=41263
--webserver_interface=127.20.120.195
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 06:55:49 dist-test-slave-gmf2 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024949, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:49.452208 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.432449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:60307 (local address 127.20.120.254:44667)
0621 06:55:49.432671 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:49.432677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:49.433818 (+ 1141us) server_negotiation.cc:407] Connection header received
0621 06:55:49.435633 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:49.435640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:49.435732 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:49.435852 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:49.437320 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.438088 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.440775 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.441230 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.441374 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.444831 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:49.444838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:49.444843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:49.444900 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:49.448663 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:49.449543 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:49.449549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:49.449553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:49.449683 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:49.450213 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:49.450218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:49.450221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:49.450578 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:49.450736 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:49.451637 (+ 901us) server_negotiation.cc:299] Negotiation successful
0621 06:55:49.451877 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 06:55:49.454408 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.432936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:60307)
0621 06:55:49.433651 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:49.433692 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:49.435381 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:49.436068 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:49.436075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:49.436620 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:55:49.437158 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:49.437174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.438319 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.438327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:49.440530 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:49.440545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.441460 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.441468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:49.441703 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.442417 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:55:49.442439 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:55:49.444557 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:55:49.448842 (+ 4285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:49.448851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:49.448868 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:49.449412 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:49.449830 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:49.449834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:49.449837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:49.450092 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:49.450836 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:55:49.450844 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:55:49.451236 (+ 392us) client_negotiation.cc:769] Sending connection context
0621 06:55:49.452629 (+ 1393us) client_negotiation.cc:240] Negotiation successful
0621 06:55:49.453137 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":257,"threads_started":1}
I20260621 06:55:49.455925 4839 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44667
I20260621 06:55:49.456244 4839 heartbeater.cc:464] Registering TS with master...
I20260621 06:55:49.456943 4839 heartbeater.cc:511] Master 127.20.120.254:44667 requested a full tablet report, sending...
I20260621 06:55:49.458531 4522 ts_manager.cc:194] Registered new tserver with Master: b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303)
I20260621 06:55:49.459748 4522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:60307
W20260621 06:55:49.784615 4842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:55:49.784896 4842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:55:49.784996 4842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:55:49.791999 4842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:55:49.792104 4842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:55:49.792224 4842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:55:49.801185 4842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:35985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=41263
--tserver_master_addrs=127.20.120.254:44667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:55:49.802452 4842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:55:49.803745 4842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:55:49.816607 4847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:49.816649 4848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:55:49.816700 4850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:55:49.819439 4842 server_base.cc:1034] running on GCE node
I20260621 06:55:49.820014 4842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:55:49.820890 4842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:55:49.822119 4842 hybrid_clock.cc:648] HybridClock initialized: now 1782024949822091 us; error 61 us; skew 500 ppm
I20260621 06:55:49.822460 4842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:55:49 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024949, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:55:49.825394 4842 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:55:49.826879 4842 webserver.cc:460] Webserver started at http://127.20.120.195:41263/ using document root <none> and password file <none>
I20260621 06:55:49.827562 4842 fs_manager.cc:359] Metadata directory not provided
I20260621 06:55:49.827718 4842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:55:49.832935 4842 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 06:55:49.835669 4857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:49.837013 4842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260621 06:55:49.837189 4842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa"
format_stamp: "Formatted at 2026-06-21 06:55:27 on dist-test-slave-gmf2"
server_key: "7448b426b1cc9e41d11fdad712cb30dd"
server_key_iv: "f78ca84ce78fe69c633ad982f242c236"
server_key_version: "encryptionkey@0"
I20260621 06:55:49.838227 4842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:55:49.855901 4842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:55:49.859658 4842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:55:49.859941 4842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:55:49.860911 4842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:55:49.862296 4842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:55:49.862442 4842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:49.862598 4842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:55:49.862699 4842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:55:49.896522 4842 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:35985
I20260621 06:55:49.896611 4969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:35985 every 8 connection(s)
I20260621 06:55:49.898021 4842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:55:49.903419 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 4842
Jun 21 06:55:49 dist-test-slave-gmf2 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024949, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:55:49.919924 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.904520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38018 (local address 127.20.120.254:44667)
0621 06:55:49.905437 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:49.905441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:49.905456 (+ 15us) server_negotiation.cc:407] Connection header received
0621 06:55:49.905497 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:49.905500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:49.905545 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:49.905626 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:49.907501 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.908225 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.910014 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.910467 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.910553 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.913525 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:49.913531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:49.913536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:49.913578 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:49.916910 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:49.917646 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:49.917654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:49.917659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:49.917820 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:49.918313 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:49.918318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:49.918322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:49.918668 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:49.918878 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:49.919241 (+ 363us) server_negotiation.cc:299] Negotiation successful
0621 06:55:49.919458 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":682,"threads_started":1}
I20260621 06:55:49.920617 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.900615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:55107 (local address 127.20.120.254:44667)
0621 06:55:49.900809 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:49.900813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:49.901974 (+ 1161us) server_negotiation.cc:407] Connection header received
0621 06:55:49.904368 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:49.904374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:49.904473 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:49.904591 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:49.906682 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.907397 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.909382 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.909796 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.909887 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.914040 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:49.914045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:49.914048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:49.914087 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:49.917117 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:49.918013 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:49.918021 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:49.918025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:49.918161 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:49.918680 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:49.918684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:49.918686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:49.919026 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:49.919164 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:49.919998 (+ 834us) server_negotiation.cc:299] Negotiation successful
0621 06:55:49.920223 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 06:55:49.921823 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.901044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:55107)
0621 06:55:49.901790 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:49.901830 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:49.904075 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:49.904876 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:49.904892 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:49.905645 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:55:49.906501 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:49.906527 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.907583 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.907587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:49.909244 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:49.909258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.910059 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.910065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:49.910286 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.911023 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:55:49.911058 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:55:49.913796 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:55:49.917249 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:49.917263 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:49.917276 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:49.917880 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:49.918307 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:55:49.918311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:55:49.918314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:55:49.918557 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:55:49.919160 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:55:49.919168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:55:49.919592 (+ 424us) client_negotiation.cc:769] Sending connection context
0621 06:55:49.919899 (+ 307us) client_negotiation.cc:240] Negotiation successful
0621 06:55:49.920432 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":275,"threads_started":1}
I20260621 06:55:49.923385 4970 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44667
I20260621 06:55:49.923703 4970 heartbeater.cc:464] Registering TS with master...
I20260621 06:55:49.924441 4970 heartbeater.cc:511] Master 127.20.120.254:44667 requested a full tablet report, sending...
I20260621 06:55:49.925788 4522 ts_manager.cc:194] Registered new tserver with Master: f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985)
I20260621 06:55:49.926606 4522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:55107
I20260621 06:55:49.935226 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:55:49.941422 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.936197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.20.120.254:44667)
0621 06:55:49.936381 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:49.936385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:49.936712 (+ 327us) server_negotiation.cc:407] Connection header received
0621 06:55:49.936898 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:49.936902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:49.937044 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:49.937169 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:55:49.937596 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.938407 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.939777 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.940161 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.940249 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.940659 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:55:49.940880 (+ 221us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0621 06:55:49.941154 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38026: token expired
Metrics: {"server-negotiator.queue_time_us":80}
W20260621 06:55:49.941562 4709 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38026: token expired
W20260621 06:55:49.942168 4442 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.120.254:44667)
I20260621 06:55:49.954392 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.942612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.20.120.254:44667)
0621 06:55:49.942767 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:49.942771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:49.942785 (+ 14us) server_negotiation.cc:407] Connection header received
0621 06:55:49.942894 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:49.942898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:49.942962 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:49.943034 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:49.944346 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.945028 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.946479 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.946874 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.946963 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.949424 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:49.949429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:49.949432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:49.949465 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:49.952111 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:49.952777 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:49.952780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:49.952783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:49.952893 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:49.953245 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:49.953250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:49.953253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:49.953535 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:49.953715 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:49.954022 (+ 307us) server_negotiation.cc:299] Negotiation successful
0621 06:55:49.954174 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 06:55:49.957607 4522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:55:49.965052 4522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:55:49.990370 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.977197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:43303 (local address 127.0.0.1:43188)
0621 06:55:49.978050 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:49.978091 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:49.978274 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:49.980114 (+ 1840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:49.980125 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:49.980166 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:49.980736 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:49.980764 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.983272 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.983281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:49.986485 (+ 3204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:49.986505 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.989054 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.989064 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:49.989361 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.989446 (+ 85us) client_negotiation.cc:769] Sending connection context
0621 06:55:49.989589 (+ 143us) client_negotiation.cc:240] Negotiation successful
0621 06:55:49.989851 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":440,"threads_started":1}
I20260621 06:55:49.991914 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.978487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.20.120.194:43303)
0621 06:55:49.979377 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:49.979386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:49.979413 (+ 27us) server_negotiation.cc:407] Connection header received
0621 06:55:49.979519 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:49.979529 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:49.979838 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:49.980997 (+ 1159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:49.981272 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.983091 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.986757 (+ 3666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.988730 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.991162 (+ 2432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.991335 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:49.991396 (+ 61us) server_negotiation.cc:299] Negotiation successful
0621 06:55:49.991563 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":625,"threads_started":1}
I20260621 06:55:49.995612 4774 tablet_service.cc:1467] Processing CreateTablet for tablet 1387ccd831684df4a8270ed9ddff131f (DEFAULT_TABLE table=test-table [id=3c87ac55dedb41fc9e9bc064d400cebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:55:49.997109 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1387ccd831684df4a8270ed9ddff131f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:49.998631 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.979202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38299 (local address 127.0.0.1:57662)
0621 06:55:49.990621 (+ 11419us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:49.990637 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:49.990728 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:49.991869 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:49.991874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:49.991905 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:49.992083 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:49.992097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.994071 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.994076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:49.996163 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:49.996179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.997783 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.997791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:49.997992 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.998030 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 06:55:49.998130 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 06:55:49.998269 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11339,"mutex_wait_us":79,"thread_start_us":13208,"threads_started":1}
I20260621 06:55:49.998934 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.979395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.20.120.193:38299)
0621 06:55:49.991130 (+ 11735us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:49.991139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:49.991166 (+ 27us) server_negotiation.cc:407] Connection header received
0621 06:55:49.991272 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:49.991280 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:49.991491 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:49.992274 (+ 783us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:49.992515 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.993873 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.996359 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.997604 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.997807 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.997990 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:49.998446 (+ 456us) server_negotiation.cc:299] Negotiation successful
0621 06:55:49.998624 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11562,"thread_start_us":12233,"threads_started":1}
I20260621 06:55:49.999008 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.977336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:35985 (local address 127.0.0.1:54330)
0621 06:55:49.978810 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:49.978826 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:49.978930 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:49.991197 (+ 12267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:49.991202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:49.991226 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:49.991582 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:49.991599 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.993801 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.993808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:49.996258 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:49.996274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.998163 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.998169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:49.998327 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.998358 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 06:55:49.998440 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 06:55:49.998559 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1342,"mutex_wait_us":185,"thread_start_us":1516,"threads_started":1}
I20260621 06:55:49.999742 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:49.978031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.20.120.195:35985)
0621 06:55:49.990723 (+ 12692us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:49.990731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:49.990757 (+ 26us) server_negotiation.cc:407] Connection header received
0621 06:55:49.990831 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:49.990836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:49.991009 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:49.991200 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:49.991843 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.993659 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.996443 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:49.997883 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:49.998153 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:49.998337 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:49.999369 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0621 06:55:49.999508 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12534,"thread_start_us":12395,"threads_started":1}
I20260621 06:55:50.002074 4642 tablet_service.cc:1467] Processing CreateTablet for tablet 1387ccd831684df4a8270ed9ddff131f (DEFAULT_TABLE table=test-table [id=3c87ac55dedb41fc9e9bc064d400cebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:55:50.002569 4905 tablet_service.cc:1467] Processing CreateTablet for tablet 1387ccd831684df4a8270ed9ddff131f (DEFAULT_TABLE table=test-table [id=3c87ac55dedb41fc9e9bc064d400cebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:55:50.003512 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1387ccd831684df4a8270ed9ddff131f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:50.003805 4905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1387ccd831684df4a8270ed9ddff131f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:55:50.012660 4707 heartbeater.cc:503] Master 127.20.120.254:44667 was elected leader, sending a full tablet report...
I20260621 06:55:50.021920 4986 tablet_bootstrap.cc:492] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6: Bootstrap starting.
I20260621 06:55:50.022162 4988 tablet_bootstrap.cc:492] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: Bootstrap starting.
I20260621 06:55:50.025049 4987 tablet_bootstrap.cc:492] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa: Bootstrap starting.
I20260621 06:55:50.025830 4986 tablet_bootstrap.cc:654] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:50.027217 4986 log.cc:824] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:50.027480 4988 tablet_bootstrap.cc:654] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:50.028434 4987 tablet_bootstrap.cc:654] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa: Neither blocks nor log segments found. Creating new log.
I20260621 06:55:50.028746 4988 log.cc:824] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:50.029639 4987 log.cc:824] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa: Log is configured to *not* fsync() on all Append() calls
I20260621 06:55:50.030584 4986 tablet_bootstrap.cc:492] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6: No bootstrap required, opened a new log
I20260621 06:55:50.030862 4986 ts_tablet_manager.cc:1397] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260621 06:55:50.031530 4988 tablet_bootstrap.cc:492] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: No bootstrap required, opened a new log
I20260621 06:55:50.031798 4988 ts_tablet_manager.cc:1397] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 06:55:50.032536 4987 tablet_bootstrap.cc:492] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa: No bootstrap required, opened a new log
I20260621 06:55:50.032788 4987 ts_tablet_manager.cc:1397] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 06:55:50.039336 4988 raft_consensus.cc:348] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:55:50.039708 4988 raft_consensus.cc:374] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:50.039841 4988 raft_consensus.cc:729] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b667d43be7a4d55bbf3e7f3a7196cf6, State: Initialized, Role: FOLLOWER
I20260621 06:55:50.040436 4988 consensus_queue.cc:260] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:55:50.041465 4986 raft_consensus.cc:348] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:55:50.041975 4986 raft_consensus.cc:374] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:50.042071 4986 raft_consensus.cc:729] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7546aef7922492fa884eb3f8241eaf6, State: Initialized, Role: FOLLOWER
I20260621 06:55:50.042654 4986 consensus_queue.cc:260] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:55:50.043078 4987 raft_consensus.cc:348] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:55:50.043504 4987 raft_consensus.cc:374] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:55:50.043602 4987 raft_consensus.cc:729] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22a2eebf84f4ae2af1b81768b5ec9fa, State: Initialized, Role: FOLLOWER
I20260621 06:55:50.044160 4987 consensus_queue.cc:260] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:55:50.044392 4986 ts_tablet_manager.cc:1428] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260621 06:55:50.044844 4839 heartbeater.cc:503] Master 127.20.120.254:44667 was elected leader, sending a full tablet report...
I20260621 06:55:50.045284 4970 heartbeater.cc:503] Master 127.20.120.254:44667 was elected leader, sending a full tablet report...
I20260621 06:55:50.045820 4987 ts_tablet_manager.cc:1428] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260621 06:55:50.046448 4988 ts_tablet_manager.cc:1428] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: Time spent starting tablet: real 0.015s user 0.011s sys 0.000s
I20260621 06:55:50.101951 4992 raft_consensus.cc:482] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:55:50.102334 4992 raft_consensus.cc:504] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:55:50.104458 4992 leader_election.cc:288] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303), f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985)
I20260621 06:55:50.112672 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.105009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:49107 (local address 127.20.120.194:43303)
0621 06:55:50.105259 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:50.105268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:50.105989 (+ 721us) server_negotiation.cc:407] Connection header received
0621 06:55:50.106221 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:50.106228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:50.106321 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:50.106420 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:50.107173 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.107952 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.110183 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.111618 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.111799 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:50.111880 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:50.112285 (+ 405us) server_negotiation.cc:299] Negotiation successful
0621 06:55:50.112415 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260621 06:55:50.112813 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.104951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:41009 (local address 127.20.120.195:35985)
0621 06:55:50.105162 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:50.105169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:50.106204 (+ 1035us) server_negotiation.cc:407] Connection header received
0621 06:55:50.106360 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:50.106364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:50.106455 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:50.106550 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:55:50.107243 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.107920 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.110404 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.111799 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.111997 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:50.112067 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:50.112443 (+ 376us) server_negotiation.cc:299] Negotiation successful
0621 06:55:50.112577 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 06:55:50.112879 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.104980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:43303 (local address 127.20.120.193:49107)
0621 06:55:50.105858 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:50.105878 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:50.106063 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:50.106449 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:50.106453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:50.106477 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:50.106988 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:50.106999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.108124 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.108128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:50.110023 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:50.110035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.111822 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.111828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:50.112031 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:50.112117 (+ 86us) client_negotiation.cc:769] Sending connection context
0621 06:55:50.112255 (+ 138us) client_negotiation.cc:240] Negotiation successful
0621 06:55:50.112419 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":121,"spinlock_wait_cycles":23936,"thread_start_us":521,"threads_started":1}
I20260621 06:55:50.113404 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.105050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:35985 (local address 127.20.120.193:41009)
0621 06:55:50.106040 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:50.106071 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 06:55:50.106233 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:55:50.106592 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:55:50.106597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:55:50.106645 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:55:50.107021 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:50.107031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.108075 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.108080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:50.110133 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:55:50.110150 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.112041 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.112047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:55:50.112271 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:50.112318 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 06:55:50.112399 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 06:55:50.112498 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":53,"thread_start_us":466,"threads_started":1}
I20260621 06:55:50.113869 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1387ccd831684df4a8270ed9ddff131f" candidate_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7546aef7922492fa884eb3f8241eaf6" is_pre_election: true
I20260621 06:55:50.114437 4794 raft_consensus.cc:2454] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b667d43be7a4d55bbf3e7f3a7196cf6 in term 0.
I20260621 06:55:50.114467 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1387ccd831684df4a8270ed9ddff131f" candidate_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" is_pre_election: true
I20260621 06:55:50.114998 4925 raft_consensus.cc:2454] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b667d43be7a4d55bbf3e7f3a7196cf6 in term 0.
I20260621 06:55:50.115034 4596 leader_election.cc:302] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b667d43be7a4d55bbf3e7f3a7196cf6, b7546aef7922492fa884eb3f8241eaf6; no voters:
I20260621 06:55:50.115372 4992 raft_consensus.cc:2781] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:55:50.115538 4992 raft_consensus.cc:482] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:55:50.115651 4992 raft_consensus.cc:3037] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:50.117854 4992 raft_consensus.cc:504] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:55:50.118475 4992 leader_election.cc:288] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 1 election: Requested vote from peers b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303), f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985)
I20260621 06:55:50.118889 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1387ccd831684df4a8270ed9ddff131f" candidate_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7546aef7922492fa884eb3f8241eaf6"
I20260621 06:55:50.118947 4925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1387ccd831684df4a8270ed9ddff131f" candidate_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa"
I20260621 06:55:50.119149 4925 raft_consensus.cc:3037] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:50.119149 4794 raft_consensus.cc:3037] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:55:50.122359 4794 raft_consensus.cc:2454] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b667d43be7a4d55bbf3e7f3a7196cf6 in term 1.
I20260621 06:55:50.122359 4925 raft_consensus.cc:2454] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b667d43be7a4d55bbf3e7f3a7196cf6 in term 1.
I20260621 06:55:50.122769 4596 leader_election.cc:302] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7b667d43be7a4d55bbf3e7f3a7196cf6, b7546aef7922492fa884eb3f8241eaf6, f22a2eebf84f4ae2af1b81768b5ec9fa; no voters:
I20260621 06:55:50.123148 4992 raft_consensus.cc:2781] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:55:50.123549 4992 raft_consensus.cc:686] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 LEADER]: Becoming Leader. State: Replica: 7b667d43be7a4d55bbf3e7f3a7196cf6, State: Running, Role: LEADER
I20260621 06:55:50.123994 4992 consensus_queue.cc:237] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:55:50.128244 4520 catalog_manager.cc:5495] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b667d43be7a4d55bbf3e7f3a7196cf6 (127.20.120.193). New cstate: current_term: 1 leader_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } health_report { overall_health: HEALTHY } } }
W20260621 06:55:50.149647 4971 tablet.cc:2369] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:55:50.153856 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.148206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.20.120.193:38299)
0621 06:55:50.148398 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:50.148403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:50.148420 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:55:50.148538 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:50.148541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:50.148622 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:50.148750 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:55:50.149052 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.150145 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.151604 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.152149 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.152287 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:50.152783 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:55:50.153281 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:55:50.153442 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:50.153501 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 06:55:50.153583 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:55:50.161825 4925 raft_consensus.cc:1261] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa [term 1 FOLLOWER]: Refusing update from remote peer 7b667d43be7a4d55bbf3e7f3a7196cf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:55:50.161825 4794 raft_consensus.cc:1261] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6 [term 1 FOLLOWER]: Refusing update from remote peer 7b667d43be7a4d55bbf3e7f3a7196cf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:55:50.162919 4997 consensus_queue.cc:1035] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:55:50.163324 4992 consensus_queue.cc:1035] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:55:50.188766 4521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:55:50.189061 4521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:55:50.192641 4521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Sending DeleteTablet for 3 replicas of tablet 1387ccd831684df4a8270ed9ddff131f
I20260621 06:55:50.193815 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet 1387ccd831684df4a8270ed9ddff131f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:55:50 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:43188
I20260621 06:55:50.193959 4905 tablet_service.cc:1514] Processing DeleteTablet for tablet 1387ccd831684df4a8270ed9ddff131f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:55:50 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:54330
I20260621 06:55:50.194013 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet 1387ccd831684df4a8270ed9ddff131f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:55:50 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:57662
I20260621 06:55:50.195322 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 4559
I20260621 06:55:50.219507 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.196278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57672 (local address 127.20.120.193:38299)
0621 06:55:50.196499 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:50.196509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:50.196625 (+ 116us) server_negotiation.cc:407] Connection header received
0621 06:55:50.196857 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:50.196865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:50.196966 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:50.197098 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:50.199781 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.201332 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.204082 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.205079 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.205226 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:50.209695 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:50.209715 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:50.209750 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:50.209811 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:50.214972 (+ 5161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:50.216039 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:50.216052 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:50.216058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:50.216287 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:50.217232 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:50.217240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:50.217246 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:50.217888 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:50.218269 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:50.218649 (+ 380us) server_negotiation.cc:299] Negotiation successful
0621 06:55:50.218977 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 06:55:50.220384 4702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:50.228325 5009 tablet_replica.cc:330] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6: stopping tablet replica
I20260621 06:55:50.228765 5009 raft_consensus.cc:2229] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:55:50.229264 5009 raft_consensus.cc:2258] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:55:50.229992 5010 tablet_replica.cc:330] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa: stopping tablet replica
I20260621 06:55:50.230477 5010 raft_consensus.cc:2229] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:55:50.231069 5010 raft_consensus.cc:2258] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:55:50.232758 5009 ts_tablet_manager.cc:1905] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:50.233105 5010 ts_tablet_manager.cc:1905] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:55:50.237596 5010 ts_tablet_manager.cc:1918] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:55:50.237769 5010 log.cc:1196] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/1387ccd831684df4a8270ed9ddff131f
I20260621 06:55:50.238260 5010 ts_tablet_manager.cc:1939] T 1387ccd831684df4a8270ed9ddff131f P f22a2eebf84f4ae2af1b81768b5ec9fa: Deleting consensus metadata
I20260621 06:55:50.239768 4509 catalog_manager.cc:4841] TS f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985): tablet 1387ccd831684df4a8270ed9ddff131f (table test-table [id=3c87ac55dedb41fc9e9bc064d400cebb]) successfully deleted
I20260621 06:55:50.239835 5009 ts_tablet_manager.cc:1918] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:55:50.239984 5009 log.cc:1196] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/1387ccd831684df4a8270ed9ddff131f
I20260621 06:55:50.240525 5009 ts_tablet_manager.cc:1939] T 1387ccd831684df4a8270ed9ddff131f P b7546aef7922492fa884eb3f8241eaf6: Deleting consensus metadata
I20260621 06:55:50.242348 4507 catalog_manager.cc:4841] TS b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303): tablet 1387ccd831684df4a8270ed9ddff131f (table test-table [id=3c87ac55dedb41fc9e9bc064d400cebb]) successfully deleted
I20260621 06:55:50.326330 5011 tablet_replica.cc:330] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: stopping tablet replica
I20260621 06:55:50.326810 5011 raft_consensus.cc:2229] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:55:50.327450 5011 raft_consensus.cc:2258] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:55:50.328790 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 4559
W20260621 06:55:50.347616 4509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.120.193:38299 (error 108)
I20260621 06:55:50.347898 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 4711
I20260621 06:55:50.349469 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.349068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38299 (local address 127.0.0.1:57682)
0621 06:55:50.349228 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:50.349328 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260621 06:55:50.349707 4509 catalog_manager.cc:4584] TS 7b667d43be7a4d55bbf3e7f3a7196cf6 (127.20.120.193:38299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1387ccd831684df4a8270ed9ddff131f: Network error: Client connection negotiation failed: client connection to 127.20.120.193:38299: connect: Connection refused (error 111)
I20260621 06:55:50.361953 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.348707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.20.120.194:43303)
0621 06:55:50.348878 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:50.348882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:50.348950 (+ 68us) server_negotiation.cc:407] Connection header received
0621 06:55:50.349168 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:50.349174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:50.349261 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:50.349363 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:50.350948 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.351815 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.353136 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.353568 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.353705 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:50.356420 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:50.356441 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:50.356455 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:50.356490 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:50.359095 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:50.359853 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:50.359860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:50.359862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:50.359984 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:50.360355 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:50.360361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:50.360364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:50.360747 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:50.361056 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:50.361341 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 06:55:50.361548 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:55:50.362644 4834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:50.403788 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.403081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38299 (local address 127.0.0.1:57688)
0621 06:55:50.403418 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:50.403551 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260621 06:55:50.404932 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.404525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38299 (local address 127.0.0.1:57694)
0621 06:55:50.404670 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:50.404769 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 06:55:50.454909 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 4711
I20260621 06:55:50.465794 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.465197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38299 (local address 127.0.0.1:57708)
0621 06:55:50.465465 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:50.465586 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260621 06:55:50.466939 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.466509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38299 (local address 127.0.0.1:57710)
0621 06:55:50.466681 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:50.466777 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 06:55:50.472931 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 4842
I20260621 06:55:50.486404 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.473658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.20.120.195:35985)
0621 06:55:50.473832 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 06:55:50.473837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:55:50.473855 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:55:50.474033 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:55:50.474037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:55:50.474109 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:55:50.474211 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:55:50.475659 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.476337 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.477609 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:55:50.478235 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:55:50.478366 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:55:50.480870 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:55:50.480883 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:55:50.480899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:55:50.480935 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:55:50.483575 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:50.484348 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:50.484355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:50.484357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:50.484493 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:55:50.484880 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:55:50.484887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:55:50.484890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:55:50.485223 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:55:50.485529 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0621 06:55:50.485839 (+ 310us) server_negotiation.cc:299] Negotiation successful
0621 06:55:50.486098 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 06:55:50.487022 4965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:50.547580 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.547017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38299 (local address 127.0.0.1:57712)
0621 06:55:50.547255 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:50.547367 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260621 06:55:50.548805 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:55:50.548380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:38299 (local address 127.0.0.1:57724)
0621 06:55:50.548607 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 06:55:50.548695 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260621 06:55:50.585039 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 4842
I20260621 06:55:50.605859 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 4489
I20260621 06:55:50.607089 4552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:55:50.720537 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 4489
I20260621 06:56:10.742679 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44667
--webserver_interface=127.20.120.254
--webserver_port=45933
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:11.081338 5017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:11.081699 5017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:11.081863 5017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:11.089183 5017 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 06:56:11.089332 5017 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 06:56:11.089426 5017 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 06:56:11.089530 5017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:56:11.089627 5017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:11.089727 5017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:11.089813 5017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:56:11.089898 5017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:56:11.099989 5017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=45933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:11.101300 5017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:11.102777 5017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:11.114735 5022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:11.114813 5025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:11.114924 5023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:11.115471 5017 server_base.cc:1034] running on GCE node
I20260621 06:56:11.116321 5017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:11.117754 5017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:11.119006 5017 hybrid_clock.cc:648] HybridClock initialized: now 1782024971118945 us; error 62 us; skew 500 ppm
I20260621 06:56:11.119357 5017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:11 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024971, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:11.125981 5017 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:56:11.127501 5017 webserver.cc:460] Webserver started at http://127.20.120.254:45933/ using document root <none> and password file <none>
I20260621 06:56:11.128161 5017 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:11.128247 5017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:11.133458 5017 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:56:11.136488 5032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:11.137894 5017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 06:56:11.138129 5017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "040658089fb548289f671cef93ae862a"
format_stamp: "Formatted at 2026-06-21 06:55:25 on dist-test-slave-gmf2"
server_key: "cdc5d7a978d69a9be7bdbc0ec6d17980"
server_key_iv: "e31f089bfc2c5a19adb76e8e1db481a5"
server_key_version: "encryptionkey@0"
I20260621 06:56:11.139031 5017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:11.154184 5017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:11.158212 5017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:11.158468 5017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:11.177315 5017 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:44667
I20260621 06:56:11.177351 5083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:44667 every 8 connection(s)
I20260621 06:56:11.178802 5017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:56:11.185606 5084 sys_catalog.cc:263] Verifying existing consensus state
I20260621 06:56:11.187731 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 5017
I20260621 06:56:11.188759 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:38299
--local_ip_for_outbound_sockets=127.20.120.193
--tserver_master_addrs=127.20.120.254:44667
--webserver_port=42475
--webserver_interface=127.20.120.193
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:56:11.189462 5084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Bootstrap starting.
I20260621 06:56:11.212252 5084 log.cc:824] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:11.228369 5084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 06:56:11.228912 5084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Bootstrap complete.
I20260621 06:56:11.237990 5084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } }
I20260621 06:56:11.238869 5084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040658089fb548289f671cef93ae862a, State: Initialized, Role: FOLLOWER
I20260621 06:56:11.239634 5084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } }
I20260621 06:56:11.239874 5084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:56:11.239984 5084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:56:11.240191 5084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 2 FOLLOWER]: Advancing to term 3
I20260621 06:56:11.244448 5084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } }
I20260621 06:56:11.244947 5084 leader_election.cc:302] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 040658089fb548289f671cef93ae862a; no voters:
I20260621 06:56:11.245666 5084 leader_election.cc:288] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 06:56:11.245800 5088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 3 FOLLOWER]: Leader election won for term 3
I20260621 06:56:11.246878 5088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [term 3 LEADER]: Becoming Leader. State: Replica: 040658089fb548289f671cef93ae862a, State: Running, Role: LEADER
I20260621 06:56:11.247483 5088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } }
I20260621 06:56:11.247822 5084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:56:11.249979 5089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "040658089fb548289f671cef93ae862a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } } }
I20260621 06:56:11.250398 5089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:11.256731 5090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040658089fb548289f671cef93ae862a. Latest consensus state: current_term: 3 leader_uuid: "040658089fb548289f671cef93ae862a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040658089fb548289f671cef93ae862a" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44667 } } }
I20260621 06:56:11.257598 5090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:11.257781 5094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:56:11.269032 5094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:56:11.271078 5094 catalog_manager.cc:1259] Loaded cluster ID: 82faa5712fe741f9aecf68c38047628a
I20260621 06:56:11.271312 5094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:56:11.276660 5094 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:56:11.279006 5094 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Loaded TSK: 0
I20260621 06:56:11.279286 5094 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 040658089fb548289f671cef93ae862a: Loaded TSK: 1
F20260621 06:56:11.280813 5094 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0c2798f9ad google::LogMessage::Fail() at ??:0
@ 0x7f0c2799288a google::LogMessage::SendToLog() at ??:0
@ 0x7f0c2798f4de google::LogMessage::Flush() at ??:0
@ 0x7f0c27990a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0c2b6f37d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0c2b6e273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f0c3d3be1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0c3d3bd783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0c3d4492a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f0c3d4490c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f0c3d4935e9 std::function<>::operator()() at ??:0
@ 0x7f0c3d3c1cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0c3d3bfc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0c3d447610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0c3d447439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0c2a5db345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0c2a5de91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0c2a5de799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0c2a5b9eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0c2d8936db start_thread at ??:0
@ 0x7f0c2682961f clone at ??:0
W20260621 06:56:11.583719 5086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:11.583994 5086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:11.584193 5086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:11.591769 5086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:11.591888 5086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:11.592096 5086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:56:11.602113 5086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:38299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=42475
--tserver_master_addrs=127.20.120.254:44667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:11.603425 5086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:11.604792 5086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:11.617367 5132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:11.617457 5134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:11.617437 5131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:11.618407 5086 server_base.cc:1034] running on GCE node
I20260621 06:56:11.618975 5086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:11.619858 5086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:11.621095 5086 hybrid_clock.cc:648] HybridClock initialized: now 1782024971621056 us; error 51 us; skew 500 ppm
I20260621 06:56:11.621436 5086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:11 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024971, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:11.624684 5086 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:56:11.626065 5086 webserver.cc:460] Webserver started at http://127.20.120.193:42475/ using document root <none> and password file <none>
I20260621 06:56:11.626714 5086 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:11.626837 5086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:11.631855 5086 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 06:56:11.634593 5141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:11.635865 5086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:56:11.636030 5086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6"
format_stamp: "Formatted at 2026-06-21 06:55:26 on dist-test-slave-gmf2"
server_key: "53338185011ee1b324c780087938c99e"
server_key_iv: "e115044e30904daa1c3090ba8799ff34"
server_key_version: "encryptionkey@0"
I20260621 06:56:11.636950 5086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:11.656368 5086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:11.660704 5086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:11.660977 5086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:11.661895 5086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:11.663731 5148 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260621 06:56:11.669188 5086 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 06:56:11.669318 5086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260621 06:56:11.669457 5086 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 06:56:11.672525 5086 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 06:56:11.672627 5086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260621 06:56:11.672933 5148 tablet_bootstrap.cc:492] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: Bootstrap starting.
I20260621 06:56:11.681198 5148 log.cc:824] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:11.695926 5148 tablet_bootstrap.cc:492] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 06:56:11.696770 5148 tablet_bootstrap.cc:492] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: Bootstrap complete.
I20260621 06:56:11.697209 5148 ts_tablet_manager.cc:1397] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.008s
I20260621 06:56:11.707659 5148 raft_consensus.cc:348] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:56:11.708777 5148 raft_consensus.cc:729] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b667d43be7a4d55bbf3e7f3a7196cf6, State: Initialized, Role: FOLLOWER
I20260621 06:56:11.709410 5148 consensus_queue.cc:260] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:56:11.711102 5148 ts_tablet_manager.cc:1428] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6: Time spent starting tablet: real 0.014s user 0.008s sys 0.008s
I20260621 06:56:11.718967 5086 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:38299
I20260621 06:56:11.719009 5256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:38299 every 8 connection(s)
I20260621 06:56:11.720172 5086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:56:11.724619 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:11.722854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:56647)
0621 06:56:11.723599 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:11.723827 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":308,"threads_started":1}
W20260621 06:56:11.725273 5257 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:44667 (0 consecutive failures): Network error: Failed to ping master at 127.20.120.254:44667: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
I20260621 06:56:11.726075 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:11.725723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:58963)
0621 06:56:11.725864 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:11.725980 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 06:56:11.726476 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 5086
I20260621 06:56:11.727021 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:11.726605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:44015)
0621 06:56:11.726771 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:11.726861 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260621 06:56:11.727275 5257 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 06:56:11.727305 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:43303
--local_ip_for_outbound_sockets=127.20.120.194
--tserver_master_addrs=127.20.120.254:44667
--webserver_port=40181
--webserver_interface=127.20.120.194
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:12.075730 5260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:12.076036 5260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:12.076262 5260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:12.084354 5260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:12.084497 5260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:12.084676 5260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:56:12.094777 5260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:43303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=40181
--tserver_master_addrs=127.20.120.254:44667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:12.096357 5260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:12.097764 5260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:12.111164 5265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:12.111330 5266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:12.111572 5268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:12.113125 5260 server_base.cc:1034] running on GCE node
I20260621 06:56:12.113802 5260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:12.114856 5260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:12.116091 5260 hybrid_clock.cc:648] HybridClock initialized: now 1782024972116027 us; error 65 us; skew 500 ppm
I20260621 06:56:12.116430 5260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:12 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024972, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:12.119814 5260 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:56:12.121304 5260 webserver.cc:460] Webserver started at http://127.20.120.194:40181/ using document root <none> and password file <none>
I20260621 06:56:12.122061 5260 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:12.122174 5260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:12.127700 5260 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 06:56:12.130378 5275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:12.131763 5260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:56:12.131948 5260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "b7546aef7922492fa884eb3f8241eaf6"
format_stamp: "Formatted at 2026-06-21 06:55:26 on dist-test-slave-gmf2"
server_key: "986ada03cc6d4910c3a200b163f277a5"
server_key_iv: "e60e425c5656eab2d52b6e6dc48fac80"
server_key_version: "encryptionkey@0"
I20260621 06:56:12.132969 5260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:12.166853 5260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:12.170991 5260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:12.171307 5260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:12.172313 5260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:12.173710 5260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:12.173861 5260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:12.174031 5260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:12.174135 5260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:12.207435 5260 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:43303
I20260621 06:56:12.207533 5387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:43303 every 8 connection(s)
I20260621 06:56:12.208998 5260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:56:12.212015 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 5260
I20260621 06:56:12.212709 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:35985
--local_ip_for_outbound_sockets=127.20.120.195
--tserver_master_addrs=127.20.120.254:44667
--webserver_port=41263
--webserver_interface=127.20.120.195
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:56:12.214423 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:12.212065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:57013)
0621 06:56:12.213000 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:12.213338 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":391,"threads_started":1}
W20260621 06:56:12.215132 5388 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:44667 (0 consecutive failures): Network error: Failed to ping master at 127.20.120.254:44667: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
I20260621 06:56:12.216027 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:12.215706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:56545)
0621 06:56:12.215857 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:12.215923 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 06:56:12.216907 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:12.216610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:54311)
0621 06:56:12.216737 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:12.216807 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 06:56:12.217154 5388 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 06:56:12.539426 5391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:12.539741 5391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:12.539906 5391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:12.547227 5391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:12.547374 5391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:12.547554 5391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:56:12.558012 5391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:35985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=41263
--tserver_master_addrs=127.20.120.254:44667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:12.559357 5391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:12.560688 5391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:12.574057 5396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:12.574023 5399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:12.573992 5397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:12.574837 5391 server_base.cc:1034] running on GCE node
I20260621 06:56:12.575428 5391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:12.576354 5391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:12.577585 5391 hybrid_clock.cc:648] HybridClock initialized: now 1782024972577534 us; error 41 us; skew 500 ppm
I20260621 06:56:12.577900 5391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:12 dist-test-slave-gmf2 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024972, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:12.580920 5391 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:56:12.582486 5391 webserver.cc:460] Webserver started at http://127.20.120.195:41263/ using document root <none> and password file <none>
I20260621 06:56:12.583202 5391 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:12.583350 5391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:12.588624 5391 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 06:56:12.591372 5406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:12.592690 5391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 06:56:12.592864 5391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa"
format_stamp: "Formatted at 2026-06-21 06:55:27 on dist-test-slave-gmf2"
server_key: "7448b426b1cc9e41d11fdad712cb30dd"
server_key_iv: "f78ca84ce78fe69c633ad982f242c236"
server_key_version: "encryptionkey@0"
I20260621 06:56:12.593869 5391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:12.622534 5391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:12.626837 5391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:12.627117 5391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:12.628142 5391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:12.629505 5391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:12.629647 5391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:12.629779 5391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:12.629868 5391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:12.663805 5391 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:35985
I20260621 06:56:12.663895 5518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:35985 every 8 connection(s)
I20260621 06:56:12.665376 5391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:56:12.668258 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 5391
I20260621 06:56:12.670537 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:12.668372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:49441)
0621 06:56:12.669249 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:12.669552 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":341,"threads_started":1}
W20260621 06:56:12.671202 5519 heartbeater.cc:644] Failed to heartbeat to 127.20.120.254:44667 (0 consecutive failures): Network error: Failed to ping master at 127.20.120.254:44667: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
I20260621 06:56:12.671963 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:12.671678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:38195)
0621 06:56:12.671806 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:12.671868 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 06:56:12.672801 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:12.672525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:53613)
0621 06:56:12.672638 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:12.672706 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 06:56:12.673048 5519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 06:56:12.728591 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:12.727836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:41575)
0621 06:56:12.728286 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:12.728377 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":210,"threads_started":1}
I20260621 06:56:13.218469 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:13.217650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:45843)
0621 06:56:13.218163 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:13.218248 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":206,"threads_started":1}
I20260621 06:56:13.412683 5525 raft_consensus.cc:482] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:56:13.413014 5525 raft_consensus.cc:504] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:56:13.414700 5525 leader_election.cc:288] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303), f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985)
Jun 21 06:56:13 dist-test-slave-gmf2 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024971, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
Jun 21 06:56:13 dist-test-slave-gmf2 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024971, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:56:13.441159 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:13.415040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:43303 (local address 127.20.120.193:39207)
0621 06:56:13.415703 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:13.415758 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:13.416451 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:13.420688 (+ 4237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:13.420692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:13.421442 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:13.422345 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:13.422368 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:13.423928 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:13.423935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:13.425680 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:13.425695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:13.427005 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:13.427011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:13.427269 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:13.428396 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:56:13.428431 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:13.431342 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:13.436415 (+ 5073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:13.436428 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:13.436445 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:13.437254 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:13.437824 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:13.437829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:13.437834 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:13.438165 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:13.439306 (+ 1141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:13.439314 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:13.439761 (+ 447us) client_negotiation.cc:769] Sending connection context
0621 06:56:13.440125 (+ 364us) client_negotiation.cc:240] Negotiation successful
0621 06:56:13.440669 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":108,"spinlock_wait_cycles":10240,"thread_start_us":1299,"threads_started":1}
I20260621 06:56:13.441702 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:13.415655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:46365 (local address 127.20.120.195:35985)
0621 06:56:13.416173 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:13.416191 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:13.417265 (+ 1074us) server_negotiation.cc:407] Connection header received
0621 06:56:13.419503 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:13.419513 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:13.419921 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:13.420245 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:13.422578 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:13.424605 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:13.426513 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:13.427615 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:13.427796 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:13.432250 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:13.432257 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:13.432269 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:13.432305 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:13.436210 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:13.437440 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:13.437451 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:13.437461 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:13.437670 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:13.438356 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:13.438362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:13.438366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:13.439144 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:13.439356 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:13.440788 (+ 1432us) server_negotiation.cc:299] Negotiation successful
0621 06:56:13.441271 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":503,"threads_started":1}
I20260621 06:56:13.441970 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:13.416048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:39207 (local address 127.20.120.194:43303)
0621 06:56:13.416658 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:13.416671 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:13.416718 (+ 47us) server_negotiation.cc:407] Connection header received
0621 06:56:13.419194 (+ 2476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:13.419205 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:13.419708 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:13.420088 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:13.422541 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:13.423727 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:13.425871 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:13.426795 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:13.427003 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:13.431726 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:13.431739 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:13.431764 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:13.431838 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:13.436195 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:13.437428 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:13.437439 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:13.437447 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:13.437670 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:13.438417 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:13.438426 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:13.438431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:13.439142 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:13.439341 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:13.441061 (+ 1720us) server_negotiation.cc:299] Negotiation successful
0621 06:56:13.441518 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":559,"threads_started":1}
I20260621 06:56:13.442348 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:13.415090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:35985 (local address 127.20.120.193:46365)
0621 06:56:13.417151 (+ 2061us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:13.417170 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:13.417369 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:13.420453 (+ 3084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:13.420466 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:13.421499 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:13.422349 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:13.422376 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:13.424795 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:13.424801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:13.426358 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:13.426372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:13.427819 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:13.427827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:13.427984 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:13.429419 (+ 1435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:56:13.429446 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:13.431964 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:13.436430 (+ 4466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:13.436438 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:13.436450 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:13.437253 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:13.437817 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:13.437824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:13.437830 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:13.438162 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:13.439424 (+ 1262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:13.439428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:13.439761 (+ 333us) client_negotiation.cc:769] Sending connection context
0621 06:56:13.440125 (+ 364us) client_negotiation.cc:240] Negotiation successful
0621 06:56:13.442030 (+ 1905us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1944,"mutex_wait_us":1468,"spinlock_wait_cycles":65920,"thread_start_us":1784,"threads_started":1}
W20260621 06:56:13.445830 5143 leader_election.cc:341] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303): Not found: Tablet not found: 1387ccd831684df4a8270ed9ddff131f
W20260621 06:56:13.446166 5145 leader_election.cc:341] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985): Not found: Tablet not found: 1387ccd831684df4a8270ed9ddff131f
I20260621 06:56:13.446385 5145 leader_election.cc:302] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b667d43be7a4d55bbf3e7f3a7196cf6; no voters: b7546aef7922492fa884eb3f8241eaf6, f22a2eebf84f4ae2af1b81768b5ec9fa
I20260621 06:56:13.446895 5525 raft_consensus.cc:2726] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 06:56:13.674602 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:13.673715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:49723)
0621 06:56:13.674219 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:13.674378 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":273,"threads_started":1}
I20260621 06:56:13.730000 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:13.729517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:47149)
0621 06:56:13.729699 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:13.729805 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260621 06:56:14.220077 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:14.219314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:53025)
0621 06:56:14.219813 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:14.219903 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":254,"threads_started":1}
I20260621 06:56:14.676194 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:14.675522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:43905)
0621 06:56:14.675943 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:14.676027 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":229,"threads_started":1}
I20260621 06:56:14.731638 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:14.730907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:58705)
0621 06:56:14.731365 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:14.731450 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":229,"threads_started":1}
I20260621 06:56:14.976298 5534 raft_consensus.cc:482] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:56:14.976480 5534 raft_consensus.cc:504] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:56:14.977157 5534 leader_election.cc:288] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303), f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985)
W20260621 06:56:14.978133 5143 leader_election.cc:341] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303): Not found: Tablet not found: 1387ccd831684df4a8270ed9ddff131f
W20260621 06:56:14.978353 5145 leader_election.cc:341] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985): Not found: Tablet not found: 1387ccd831684df4a8270ed9ddff131f
I20260621 06:56:14.978463 5145 leader_election.cc:302] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b667d43be7a4d55bbf3e7f3a7196cf6; no voters: b7546aef7922492fa884eb3f8241eaf6, f22a2eebf84f4ae2af1b81768b5ec9fa
I20260621 06:56:14.978662 5534 raft_consensus.cc:2726] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 06:56:15.221665 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:15.220924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:43235)
0621 06:56:15.221399 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:15.221488 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":256,"threads_started":1}
I20260621 06:56:15.677781 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:15.677084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:49125)
0621 06:56:15.677552 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:15.677630 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":258,"threads_started":1}
I20260621 06:56:15.733219 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:15.732563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:35497)
0621 06:56:15.732989 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:15.733059 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":248,"threads_started":1}
I20260621 06:56:16.223448 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:16.222712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:40303)
0621 06:56:16.223180 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:16.223270 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260621 06:56:16.679431 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:16.678653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:37557)
0621 06:56:16.679125 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:16.679225 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":233,"threads_started":1}
I20260621 06:56:16.735077 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:16.734280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:60367)
0621 06:56:16.734792 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:16.734883 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":233,"threads_started":1}
I20260621 06:56:17.058981 5541 raft_consensus.cc:482] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:56:17.059160 5541 raft_consensus.cc:504] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:56:17.059749 5541 leader_election.cc:288] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303), f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985)
W20260621 06:56:17.060887 5143 leader_election.cc:341] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303): Not found: Tablet not found: 1387ccd831684df4a8270ed9ddff131f
W20260621 06:56:17.061112 5145 leader_election.cc:341] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985): Not found: Tablet not found: 1387ccd831684df4a8270ed9ddff131f
I20260621 06:56:17.061221 5145 leader_election.cc:302] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b667d43be7a4d55bbf3e7f3a7196cf6; no voters: b7546aef7922492fa884eb3f8241eaf6, f22a2eebf84f4ae2af1b81768b5ec9fa
I20260621 06:56:17.061446 5541 raft_consensus.cc:2726] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 06:56:17.225133 5542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:17.224405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:35657)
0621 06:56:17.224872 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:17.224954 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260621 06:56:17.680924 5543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:17.680279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:42795)
0621 06:56:17.680695 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:17.680769 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":240,"threads_started":1}
I20260621 06:56:17.736572 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:17.735944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:57339)
0621 06:56:17.736326 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:17.736416 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":205,"threads_started":1}
I20260621 06:56:18.226763 5545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:18.226051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:45129)
0621 06:56:18.226500 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:18.226584 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":223,"threads_started":1}
I20260621 06:56:18.682654 5546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:18.681791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:46891)
0621 06:56:18.682293 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:18.682433 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":301,"threads_started":1}
I20260621 06:56:18.738142 5547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:18.737380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:55943)
0621 06:56:18.737837 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:18.737953 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":247,"threads_started":1}
I20260621 06:56:19.228422 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:19.227634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:57921)
0621 06:56:19.228137 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:19.228225 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":228,"threads_started":1}
I20260621 06:56:19.684352 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:19.683536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:50845)
0621 06:56:19.684086 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:19.684167 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":259,"threads_started":1}
I20260621 06:56:19.739699 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:19.738985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:60669)
0621 06:56:19.739451 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:19.739535 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":230,"threads_started":1}
I20260621 06:56:19.950675 5551 raft_consensus.cc:482] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:56:19.950923 5551 raft_consensus.cc:504] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:56:19.951670 5551 leader_election.cc:288] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303), f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985)
W20260621 06:56:19.952632 5145 leader_election.cc:341] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985): Not found: Tablet not found: 1387ccd831684df4a8270ed9ddff131f
W20260621 06:56:19.952800 5143 leader_election.cc:341] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303): Not found: Tablet not found: 1387ccd831684df4a8270ed9ddff131f
I20260621 06:56:19.952946 5143 leader_election.cc:302] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b667d43be7a4d55bbf3e7f3a7196cf6; no voters: b7546aef7922492fa884eb3f8241eaf6, f22a2eebf84f4ae2af1b81768b5ec9fa
I20260621 06:56:19.953176 5551 raft_consensus.cc:2726] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 06:56:20.229969 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:20.229280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:56999)
0621 06:56:20.229684 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:20.229769 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":220,"threads_started":1}
I20260621 06:56:20.686014 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:20.685321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:37621)
0621 06:56:20.685734 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:20.685819 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":212,"threads_started":1}
I20260621 06:56:20.741369 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:20.740589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:51481)
0621 06:56:20.741056 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:20.741143 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":224,"threads_started":1}
I20260621 06:56:21.231676 5555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:21.230914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:45569)
0621 06:56:21.231423 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:21.231510 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":248,"threads_started":1}
I20260621 06:56:21.687752 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:21.686983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:37521)
0621 06:56:21.687440 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:21.687549 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":215,"threads_started":1}
I20260621 06:56:21.743137 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:21.742413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:40269)
0621 06:56:21.742879 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:21.742971 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":242,"threads_started":1}
I20260621 06:56:22.233364 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:22.232610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:37101)
0621 06:56:22.233074 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:22.233158 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":250,"threads_started":1}
I20260621 06:56:22.689711 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:22.688764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:52015)
0621 06:56:22.689327 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:22.689426 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":308,"threads_started":1}
I20260621 06:56:22.744817 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:22.744099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:36283)
0621 06:56:22.744548 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:22.744647 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":243,"threads_started":1}
I20260621 06:56:23.235204 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:23.234435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:33133)
0621 06:56:23.234941 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:23.235042 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":268,"threads_started":1}
I20260621 06:56:23.691346 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:23.690637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:38667)
0621 06:56:23.691097 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:23.691179 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
I20260621 06:56:23.746649 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:23.745802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:48641)
0621 06:56:23.746288 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:23.746429 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":287,"threads_started":1}
I20260621 06:56:24.237066 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:24.236271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:46583)
0621 06:56:24.236764 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:24.236852 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":262,"threads_started":1}
I20260621 06:56:24.692858 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:24.692173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:43231)
0621 06:56:24.692587 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:24.692673 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":225,"threads_started":1}
I20260621 06:56:24.748093 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:24.747459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:40065)
0621 06:56:24.747858 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:24.747936 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":203,"threads_started":1}
I20260621 06:56:25.238773 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:25.238047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:42165)
0621 06:56:25.238509 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:25.238604 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":240,"threads_started":1}
I20260621 06:56:25.656934 5568 raft_consensus.cc:482] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:56:25.657159 5568 raft_consensus.cc:504] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7546aef7922492fa884eb3f8241eaf6" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 43303 } } peers { permanent_uuid: "f22a2eebf84f4ae2af1b81768b5ec9fa" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 35985 } } peers { permanent_uuid: "7b667d43be7a4d55bbf3e7f3a7196cf6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 38299 } }
I20260621 06:56:25.657804 5568 leader_election.cc:288] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303), f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985)
W20260621 06:56:25.658732 5145 leader_election.cc:341] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f22a2eebf84f4ae2af1b81768b5ec9fa (127.20.120.195:35985): Not found: Tablet not found: 1387ccd831684df4a8270ed9ddff131f
W20260621 06:56:25.658912 5143 leader_election.cc:341] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7546aef7922492fa884eb3f8241eaf6 (127.20.120.194:43303): Not found: Tablet not found: 1387ccd831684df4a8270ed9ddff131f
I20260621 06:56:25.659046 5143 leader_election.cc:302] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b667d43be7a4d55bbf3e7f3a7196cf6; no voters: b7546aef7922492fa884eb3f8241eaf6, f22a2eebf84f4ae2af1b81768b5ec9fa
I20260621 06:56:25.659314 5568 raft_consensus.cc:2726] T 1387ccd831684df4a8270ed9ddff131f P 7b667d43be7a4d55bbf3e7f3a7196cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 06:56:25.694432 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:25.693703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:52019)
0621 06:56:25.694169 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:25.694272 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":273,"threads_started":1}
I20260621 06:56:25.749740 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:25.748915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:41327)
0621 06:56:25.749393 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:25.749505 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":212,"threads_started":1}
I20260621 06:56:26.240612 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:26.239881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:35161)
0621 06:56:26.240339 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:26.240419 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":257,"threads_started":1}
I20260621 06:56:26.696116 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:26.695360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:54091)
0621 06:56:26.695827 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:26.695922 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":227,"threads_started":1}
I20260621 06:56:26.751745 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:26.751005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.193:38749)
0621 06:56:26.751462 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:26.751543 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":222,"threads_started":1}
I20260621 06:56:27.242733 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:27.241972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.194:58343)
0621 06:56:27.242450 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:27.242542 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":225,"threads_started":1}
I20260621 06:56:27.669628 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 5086
I20260621 06:56:27.689287 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:27.670780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.20.120.193:38299)
0621 06:56:27.671405 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:27.671411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:27.671435 (+ 24us) server_negotiation.cc:407] Connection header received
0621 06:56:27.671495 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:27.671501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:27.671711 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:27.671879 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:27.673319 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:27.674357 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:27.675588 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:27.676119 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:27.676251 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:27.683882 (+ 7631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:27.683893 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:27.683906 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:27.683938 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:27.686580 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:27.687249 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:27.687256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:27.687259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:27.687399 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:27.687764 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:27.687769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:27.687771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:27.688089 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:27.688387 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:27.688721 (+ 334us) server_negotiation.cc:299] Negotiation successful
0621 06:56:27.688974 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":322,"threads_started":1}
I20260621 06:56:27.690727 5252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:27.698117 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:27.697319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44667 (local address 127.20.120.195:47713)
0621 06:56:27.697784 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:27.697905 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.254:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":222,"threads_started":1}
I20260621 06:56:27.780369 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 5086
I20260621 06:56:27.797744 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 5260
I20260621 06:56:27.810614 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:27.798679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.20.120.194:43303)
0621 06:56:27.799193 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:27.799198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:27.799217 (+ 19us) server_negotiation.cc:407] Connection header received
0621 06:56:27.799279 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:27.799283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:27.799340 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:27.799429 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:27.800700 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:27.801534 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:27.802741 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:27.803171 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:27.803261 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:27.805853 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:27.805857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:27.805861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:27.805889 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:27.808270 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:27.808874 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:27.808877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:27.808879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:27.809012 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:27.809336 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:27.809339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:27.809341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:27.809562 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:27.809745 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:27.810073 (+ 328us) server_negotiation.cc:299] Negotiation successful
0621 06:56:27.810240 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":257,"threads_started":1}
I20260621 06:56:27.811335 5383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:27.893357 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 5260
I20260621 06:56:27.911304 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 5391
I20260621 06:56:27.924870 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:27.912227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.20.120.195:35985)
0621 06:56:27.912839 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:27.912843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:27.912861 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:56:27.912927 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:27.912931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:27.912993 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:27.913083 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:27.914706 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:27.915503 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:27.916666 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:27.917078 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:27.917161 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:27.919734 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:27.919739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:27.919742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:27.919774 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:27.922731 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:27.923306 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:27.923310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:27.923312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:27.923443 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:27.923755 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:27.923758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:27.923761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:27.923995 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:27.924178 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:27.924401 (+ 223us) server_negotiation.cc:299] Negotiation successful
0621 06:56:27.924566 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":320,"threads_started":1}
I20260621 06:56:27.925591 5514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:28.007611 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 5391
2026-06-21T06:56:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64124 ms)
[----------] 30 tests from SecurityITest (287336 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:56:28 dist-test-slave-gmf2 krb5kdc[5585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:56:28 dist-test-slave-gmf2 krb5kdc[5585](info): set up 2 sockets
Jun 21 06:56:28 dist-test-slave-gmf2 krb5kdc[5585](info): commencing operation
krb5kdc: starting...
W20260621 06:56:29.097944 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:56:29 dist-test-slave-gmf2 krb5kdc[5585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:56:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:56:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:56:29.339390 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:41795
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:41795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:29.664072 5598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:29.664345 5598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:29.664529 5598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:29.671202 5598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:56:29.671329 5598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:29.671424 5598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:29.671523 5598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:56:29.671607 5598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:56:29.680482 5598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:41795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:41795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:29.681680 5598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:29.682942 5598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:29.693876 5603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:29.693993 5606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:29.694022 5604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:29.694492 5598 server_base.cc:1034] running on GCE node
I20260621 06:56:29.695204 5598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:29.696465 5598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:29.697672 5598 hybrid_clock.cc:648] HybridClock initialized: now 1782024989697620 us; error 42 us; skew 500 ppm
I20260621 06:56:29.698069 5598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:29 dist-test-slave-gmf2 krb5kdc[5585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024989, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:29.701078 5598 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:56:29.702448 5598 webserver.cc:460] Webserver started at http://127.20.120.254:36919/ using document root <none> and password file <none>
I20260621 06:56:29.703090 5598 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:29.703238 5598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:29.703505 5598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:29.706301 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "6b685a3fd5e14db596ea557979c9d092"
format_stamp: "Formatted at 2026-06-21 06:56:29 on dist-test-slave-gmf2"
server_key: "0de036577eafc3c1d984d913d178dac1"
server_key_iv: "33660cbaf861bfde79cfb7145994b935"
server_key_version: "encryptionkey@0"
I20260621 06:56:29.707039 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "6b685a3fd5e14db596ea557979c9d092"
format_stamp: "Formatted at 2026-06-21 06:56:29 on dist-test-slave-gmf2"
server_key: "0de036577eafc3c1d984d913d178dac1"
server_key_iv: "33660cbaf861bfde79cfb7145994b935"
server_key_version: "encryptionkey@0"
I20260621 06:56:29.711884 5598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 06:56:29.714980 5613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:29.716199 5598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 06:56:29.716377 5598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "6b685a3fd5e14db596ea557979c9d092"
format_stamp: "Formatted at 2026-06-21 06:56:29 on dist-test-slave-gmf2"
server_key: "0de036577eafc3c1d984d913d178dac1"
server_key_iv: "33660cbaf861bfde79cfb7145994b935"
server_key_version: "encryptionkey@0"
I20260621 06:56:29.716580 5598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:29.729844 5598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:29.734017 5598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:29.734290 5598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:29.752194 5598 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:41795
I20260621 06:56:29.752213 5664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:41795 every 8 connection(s)
I20260621 06:56:29.753439 5598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:56:29.756803 5665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:29.762272 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 5598
I20260621 06:56:29.762485 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:56:29.763113 20963 external_mini_cluster.cc:1428] Setting key 27ca1c7d5485e9ebf3aef339fb52f0eb
I20260621 06:56:29.766914 5665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092: Bootstrap starting.
I20260621 06:56:29.770540 5665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:29.771596 5665 log.cc:824] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:29.774374 5665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092: No bootstrap required, opened a new log
Jun 21 06:56:29 dist-test-slave-gmf2 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:56:29.783131 5665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b685a3fd5e14db596ea557979c9d092" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 41795 } }
I20260621 06:56:29.783528 5665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:29.783653 5665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b685a3fd5e14db596ea557979c9d092, State: Initialized, Role: FOLLOWER
I20260621 06:56:29.784309 5665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b685a3fd5e14db596ea557979c9d092" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 41795 } }
I20260621 06:56:29.784610 5665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:56:29.784766 5665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:56:29.784972 5665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:29.786085 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:29.764956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.20.120.254:41795)
0621 06:56:29.765674 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:29.765683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:29.765723 (+ 40us) server_negotiation.cc:407] Connection header received
0621 06:56:29.767362 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:29.767383 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:29.767752 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:29.768032 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:29.769831 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:29.771250 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:29.772905 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:29.773528 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:29.773657 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:29.778533 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:29.778543 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:29.778555 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:29.778586 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:29.781629 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:29.782436 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:29.782443 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:29.782449 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:29.782601 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:29.783001 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:29.783007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:29.783010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:29.783507 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:29.783809 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:29.784286 (+ 477us) server_negotiation.cc:299] Negotiation successful
0621 06:56:29.784731 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":356,"threads_started":1}
I20260621 06:56:29.787654 5665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b685a3fd5e14db596ea557979c9d092" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 41795 } }
I20260621 06:56:29.788172 5665 leader_election.cc:302] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b685a3fd5e14db596ea557979c9d092; no voters:
I20260621 06:56:29.788826 5665 leader_election.cc:288] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:56:29.788880 5670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:29.789638 5670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [term 1 LEADER]: Becoming Leader. State: Replica: 6b685a3fd5e14db596ea557979c9d092, State: Running, Role: LEADER
I20260621 06:56:29.790230 5670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b685a3fd5e14db596ea557979c9d092" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 41795 } }
I20260621 06:56:29.790959 5665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:56:29.792953 5671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b685a3fd5e14db596ea557979c9d092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b685a3fd5e14db596ea557979c9d092" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 41795 } } }
I20260621 06:56:29.793437 5671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:29.794786 5672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b685a3fd5e14db596ea557979c9d092. Latest consensus state: current_term: 1 leader_uuid: "6b685a3fd5e14db596ea557979c9d092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b685a3fd5e14db596ea557979c9d092" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 41795 } } }
I20260621 06:56:29.795265 5672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:29.799220 5679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:56:29.803644 5679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:56:29.811782 5679 catalog_manager.cc:1347] Generated new cluster ID: 905df58021a94927b368b612305bd0b7
I20260621 06:56:29.811883 5679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:56:29.827597 5679 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:56:29.829345 5679 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:56:29.840574 5679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092: Generated new TSK 0
I20260621 06:56:29.841555 5679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:56:29.894271 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:41795
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:30.214753 5691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:30.215021 5691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:30.215126 5691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:30.222530 5691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:30.222640 5691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:30.222751 5691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:56:30.232520 5691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:41795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:30.233794 5691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:30.235211 5691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:30.248498 5696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:30.248557 5699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:30.248579 5697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:30.249248 5691 server_base.cc:1034] running on GCE node
I20260621 06:56:30.249897 5691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:30.250924 5691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:30.252143 5691 hybrid_clock.cc:648] HybridClock initialized: now 1782024990252082 us; error 51 us; skew 500 ppm
I20260621 06:56:30.252454 5691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:30 dist-test-slave-gmf2 krb5kdc[5585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024990, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:30.255687 5691 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:56:30.257158 5691 webserver.cc:460] Webserver started at http://127.20.120.193:41371/ using document root <none> and password file <none>
I20260621 06:56:30.257776 5691 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:30.257871 5691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:30.258196 5691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:30.260910 5691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "49843cd089fb42c796496110920209ea"
format_stamp: "Formatted at 2026-06-21 06:56:30 on dist-test-slave-gmf2"
server_key: "6d9e27652b58fab93e39de84a3b8c5fd"
server_key_iv: "57b3c3be00763aca4230960c730f7c74"
server_key_version: "encryptionkey@0"
I20260621 06:56:30.261598 5691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "49843cd089fb42c796496110920209ea"
format_stamp: "Formatted at 2026-06-21 06:56:30 on dist-test-slave-gmf2"
server_key: "6d9e27652b58fab93e39de84a3b8c5fd"
server_key_iv: "57b3c3be00763aca4230960c730f7c74"
server_key_version: "encryptionkey@0"
I20260621 06:56:30.266752 5691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 06:56:30.270063 5706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:30.271332 5691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:56:30.271458 5691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "49843cd089fb42c796496110920209ea"
format_stamp: "Formatted at 2026-06-21 06:56:30 on dist-test-slave-gmf2"
server_key: "6d9e27652b58fab93e39de84a3b8c5fd"
server_key_iv: "57b3c3be00763aca4230960c730f7c74"
server_key_version: "encryptionkey@0"
I20260621 06:56:30.271626 5691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:30.289558 5691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:30.294123 5691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:30.294355 5691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:30.295320 5691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:30.296628 5691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:30.296700 5691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:30.296787 5691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:30.296828 5691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:30.328760 5691 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:36625
I20260621 06:56:30.328805 5818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:36625 every 8 connection(s)
I20260621 06:56:30.330009 5691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:56:30.337895 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 5691
I20260621 06:56:30.338119 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:56:30.338798 20963 external_mini_cluster.cc:1428] Setting key 47b40d4f0172d0931413f4ae8992efd7
Jun 21 06:56:30 dist-test-slave-gmf2 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024990, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:56:30.351837 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:30.332501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:55591 (local address 127.20.120.254:41795)
0621 06:56:30.332947 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:30.332953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:30.333944 (+ 991us) server_negotiation.cc:407] Connection header received
0621 06:56:30.335630 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:30.335634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:30.335696 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:30.335788 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:30.337156 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:30.338225 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:30.339918 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:30.340564 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:30.340896 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:30.344838 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:30.344863 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:30.344870 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:30.344913 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:30.348574 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:30.349439 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:30.349448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:30.349454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:30.349601 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:30.350129 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:30.350134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:30.350139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:30.350414 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:30.350594 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:30.351246 (+ 652us) server_negotiation.cc:299] Negotiation successful
0621 06:56:30.351468 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":205,"threads_started":1}
I20260621 06:56:30.352720 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:30.333094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:41795 (local address 127.20.120.193:55591)
0621 06:56:30.333761 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:30.333802 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:30.335400 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:30.335988 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:30.335995 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:30.336495 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:30.337002 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:30.337019 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:30.338350 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:30.338354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:30.339735 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:30.339746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:30.340992 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:30.341000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:30.341298 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:30.342033 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:56:30.342072 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:30.344640 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:30.348726 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:30.348736 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:30.348753 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:30.349301 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:30.349712 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:30.349719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:30.349726 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:30.349987 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:30.350545 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:30.350554 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:30.350896 (+ 342us) client_negotiation.cc:769] Sending connection context
0621 06:56:30.351154 (+ 258us) client_negotiation.cc:240] Negotiation successful
0621 06:56:30.351649 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":271,"threads_started":1}
I20260621 06:56:30.354295 5819 heartbeater.cc:347] Connected to a master server at 127.20.120.254:41795
I20260621 06:56:30.354588 5819 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:30.355289 5819 heartbeater.cc:511] Master 127.20.120.254:41795 requested a full tablet report, sending...
I20260621 06:56:30.357487 5630 ts_manager.cc:194] Registered new tserver with Master: 49843cd089fb42c796496110920209ea (127.20.120.193:36625)
I20260621 06:56:30.359606 5630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:55591
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:56:30.392341 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:41795
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:30.711232 5825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:30.711547 5825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:30.711711 5825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:30.718650 5825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:30.718802 5825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:30.718972 5825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:56:30.728565 5825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:41795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:30.729837 5825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:30.731158 5825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:30.743635 5830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:30.743781 5833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:30.743804 5831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:30.744647 5825 server_base.cc:1034] running on GCE node
I20260621 06:56:30.745232 5825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:30.746275 5825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:30.747525 5825 hybrid_clock.cc:648] HybridClock initialized: now 1782024990747470 us; error 45 us; skew 500 ppm
I20260621 06:56:30.747833 5825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:30 dist-test-slave-gmf2 krb5kdc[5585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024990, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:30.750896 5825 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:56:30.752521 5825 webserver.cc:460] Webserver started at http://127.20.120.194:41651/ using document root <none> and password file <none>
I20260621 06:56:30.753301 5825 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:30.753428 5825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:30.753762 5825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:30.757406 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "c9201885dcab4257992a803668208ff2"
format_stamp: "Formatted at 2026-06-21 06:56:30 on dist-test-slave-gmf2"
server_key: "50a0e90247ff3cb5b066e00cd55574aa"
server_key_iv: "95817c3f2c9c66b09197c77f61915fc3"
server_key_version: "encryptionkey@0"
I20260621 06:56:30.758339 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "c9201885dcab4257992a803668208ff2"
format_stamp: "Formatted at 2026-06-21 06:56:30 on dist-test-slave-gmf2"
server_key: "50a0e90247ff3cb5b066e00cd55574aa"
server_key_iv: "95817c3f2c9c66b09197c77f61915fc3"
server_key_version: "encryptionkey@0"
I20260621 06:56:30.764457 5825 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260621 06:56:30.768533 5840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:30.770007 5825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:56:30.770159 5825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "c9201885dcab4257992a803668208ff2"
format_stamp: "Formatted at 2026-06-21 06:56:30 on dist-test-slave-gmf2"
server_key: "50a0e90247ff3cb5b066e00cd55574aa"
server_key_iv: "95817c3f2c9c66b09197c77f61915fc3"
server_key_version: "encryptionkey@0"
I20260621 06:56:30.770344 5825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:30.809237 5825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:30.813534 5825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:30.813789 5825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:30.814682 5825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:30.815898 5825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:30.815969 5825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:30.816048 5825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:30.816092 5825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:30.848438 5825 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:37783
I20260621 06:56:30.848526 5952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:37783 every 8 connection(s)
I20260621 06:56:30.850134 5825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:56:30.853669 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 5825
I20260621 06:56:30.853864 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:56:30.854406 20963 external_mini_cluster.cc:1428] Setting key 7a8ac3286dd5169f9a4cca26ff7f5e80
Jun 21 06:56:30 dist-test-slave-gmf2 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024990, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:56:30.870852 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:30.852576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:43955 (local address 127.20.120.254:41795)
0621 06:56:30.854160 (+ 1584us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:30.854166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:30.854870 (+ 704us) server_negotiation.cc:407] Connection header received
0621 06:56:30.856737 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:30.856743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:30.856825 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:30.856946 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:30.858454 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:30.859533 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:30.860994 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:30.861774 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:30.861943 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:30.865287 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:30.865293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:30.865298 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:30.865337 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:30.867778 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:30.868501 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:30.868506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:30.868510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:30.868644 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:30.869062 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:30.869067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:30.869072 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:30.869336 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:30.869495 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:30.870257 (+ 762us) server_negotiation.cc:299] Negotiation successful
0621 06:56:30.870482 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1429,"thread_start_us":226,"threads_started":1}
I20260621 06:56:30.871827 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:30.853152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:41795 (local address 127.20.120.194:43955)
0621 06:56:30.854696 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:30.854740 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:30.856458 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:30.857161 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:30.857169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:30.857650 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:30.858296 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:30.858314 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:30.859671 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:30.859675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:30.860855 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:30.860866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:30.862019 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:30.862024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:30.862224 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:30.862917 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:56:30.862937 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:30.865086 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:30.867914 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:30.867922 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:30.867937 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:30.868371 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:30.868752 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:30.868757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:30.868761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:30.868945 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:30.869469 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:30.869477 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:30.869816 (+ 339us) client_negotiation.cc:769] Sending connection context
0621 06:56:30.870130 (+ 314us) client_negotiation.cc:240] Negotiation successful
0621 06:56:30.870669 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":1065,"threads_started":1}
I20260621 06:56:30.873221 5953 heartbeater.cc:347] Connected to a master server at 127.20.120.254:41795
I20260621 06:56:30.873543 5953 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:30.874459 5953 heartbeater.cc:511] Master 127.20.120.254:41795 requested a full tablet report, sending...
I20260621 06:56:30.876157 5630 ts_manager.cc:194] Registered new tserver with Master: c9201885dcab4257992a803668208ff2 (127.20.120.194:37783)
I20260621 06:56:30.877008 5630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:43955
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:56:30.910274 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:41795
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:31.226716 5959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:31.226972 5959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:31.227058 5959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:31.233664 5959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:31.233752 5959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:31.233853 5959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:56:31.243134 5959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:41795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:31.244218 5959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:31.245551 5959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:31.258692 5967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:31.258790 5964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:31.258809 5965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:31.260013 5959 server_base.cc:1034] running on GCE node
I20260621 06:56:31.260668 5959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:31.261552 5959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:31.262769 5959 hybrid_clock.cc:648] HybridClock initialized: now 1782024991262733 us; error 60 us; skew 500 ppm
I20260621 06:56:31.263091 5959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:31 dist-test-slave-gmf2 krb5kdc[5585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024991, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:31.266029 5959 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:56:31.267297 5959 webserver.cc:460] Webserver started at http://127.20.120.195:42651/ using document root <none> and password file <none>
I20260621 06:56:31.268050 5959 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:31.268191 5959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:31.268486 5959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:31.271520 5959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "8aca594064a749008c4ae5c96f038f87"
format_stamp: "Formatted at 2026-06-21 06:56:31 on dist-test-slave-gmf2"
server_key: "6cf9a4281945475850170cf1cf575028"
server_key_iv: "362b333db4fd8329e0e3fb602fc64fe5"
server_key_version: "encryptionkey@0"
I20260621 06:56:31.272246 5959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "8aca594064a749008c4ae5c96f038f87"
format_stamp: "Formatted at 2026-06-21 06:56:31 on dist-test-slave-gmf2"
server_key: "6cf9a4281945475850170cf1cf575028"
server_key_iv: "362b333db4fd8329e0e3fb602fc64fe5"
server_key_version: "encryptionkey@0"
I20260621 06:56:31.277426 5959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 06:56:31.280545 5974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:31.281803 5959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 06:56:31.281987 5959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "8aca594064a749008c4ae5c96f038f87"
format_stamp: "Formatted at 2026-06-21 06:56:31 on dist-test-slave-gmf2"
server_key: "6cf9a4281945475850170cf1cf575028"
server_key_iv: "362b333db4fd8329e0e3fb602fc64fe5"
server_key_version: "encryptionkey@0"
I20260621 06:56:31.282184 5959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:31.296545 5959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:31.300613 5959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:31.300875 5959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:31.301738 5959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:31.302999 5959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:31.303097 5959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:31.303237 5959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:31.303306 5959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:31.334532 5959 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:46213
I20260621 06:56:31.334547 6086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:46213 every 8 connection(s)
I20260621 06:56:31.335898 5959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:56:31.341997 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 5959
I20260621 06:56:31.342181 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:56:31.342589 20963 external_mini_cluster.cc:1428] Setting key 46d38e02336f6d727a3d26dbe57d7a02
Jun 21 06:56:31 dist-test-slave-gmf2 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024991, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:56:31.354493 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.338185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:49751 (local address 127.20.120.254:41795)
0621 06:56:31.338351 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:31.338355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:31.339361 (+ 1006us) server_negotiation.cc:407] Connection header received
0621 06:56:31.341079 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:31.341084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:31.341199 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:31.341310 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:31.342989 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.343906 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.345671 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.346155 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.346287 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.349314 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:31.349320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:31.349325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:31.349366 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:31.351739 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:31.352377 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:31.352384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:31.352388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:31.352507 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:31.352900 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:31.352905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:31.352909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:31.353200 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:31.353361 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:31.353963 (+ 602us) server_negotiation.cc:299] Negotiation successful
0621 06:56:31.354187 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 06:56:31.355320 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.338566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:41795 (local address 127.20.120.195:49751)
0621 06:56:31.339227 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:31.339265 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:31.340827 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:31.341542 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:31.341550 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:31.342141 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:31.342809 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:31.342828 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.344048 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.344053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:31.345522 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:31.345534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.346362 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.346369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:31.346588 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.347205 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:56:31.347225 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:31.349133 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:31.351861 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:31.351869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:31.351882 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:31.352265 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:31.352619 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:31.352624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:31.352627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:31.352792 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:31.353321 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:31.353330 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:31.353631 (+ 301us) client_negotiation.cc:769] Sending connection context
0621 06:56:31.353871 (+ 240us) client_negotiation.cc:240] Negotiation successful
0621 06:56:31.354323 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":267,"threads_started":1}
I20260621 06:56:31.356789 6087 heartbeater.cc:347] Connected to a master server at 127.20.120.254:41795
I20260621 06:56:31.357115 6087 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:31.357846 6087 heartbeater.cc:511] Master 127.20.120.254:41795 requested a full tablet report, sending...
I20260621 06:56:31.359193 5629 ts_manager.cc:194] Registered new tserver with Master: 8aca594064a749008c4ae5c96f038f87 (127.20.120.195:46213)
I20260621 06:56:31.360052 5629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:49751
I20260621 06:56:31.362416 5819 heartbeater.cc:503] Master 127.20.120.254:41795 was elected leader, sending a full tablet report...
I20260621 06:56:31.369563 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:56:31.385269 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.373781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.20.120.254:41795)
0621 06:56:31.373977 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:31.373981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:31.374375 (+ 394us) server_negotiation.cc:407] Connection header received
0621 06:56:31.374570 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:31.374574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:31.374646 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:31.374721 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:31.376178 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.376850 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.378132 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.378484 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.378565 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.380714 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:31.380718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:31.380722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:31.380748 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:31.383148 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:31.383737 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:31.383741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:31.383743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:31.383852 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:31.384136 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:31.384139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:31.384141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:31.384394 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:31.384569 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:31.384881 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 06:56:31.385052 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 06:56:31.388828 5629 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:56:31.395435 5629 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:56:31.416738 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.407359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.20.120.193:36625)
0621 06:56:31.410453 (+ 3094us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:31.410461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:31.410486 (+ 25us) server_negotiation.cc:407] Connection header received
0621 06:56:31.410577 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:31.410584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:31.410789 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:31.410957 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:31.411648 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.412866 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.414538 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.415557 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.415792 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.415960 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:31.416226 (+ 266us) server_negotiation.cc:299] Negotiation successful
0621 06:56:31.416384 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2921,"thread_start_us":2762,"threads_started":1}
I20260621 06:56:31.416738 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.406641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:36625 (local address 127.0.0.1:37948)
0621 06:56:31.408433 (+ 1792us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:31.408448 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:31.408575 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:31.410985 (+ 2410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:31.410992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:31.411029 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:31.411488 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:31.411500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.412978 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.412984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:31.414385 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:31.414399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.415781 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.415787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:31.415978 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.416029 (+ 51us) client_negotiation.cc:769] Sending connection context
0621 06:56:31.416186 (+ 157us) client_negotiation.cc:240] Negotiation successful
0621 06:56:31.416373 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1675,"thread_start_us":893,"threads_started":1}
I20260621 06:56:31.418009 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.407834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:37783 (local address 127.0.0.1:41278)
0621 06:56:31.408569 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:31.408598 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:31.408704 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:31.409644 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:31.409648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:31.409670 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:31.409961 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:31.409975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.411890 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.411894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:31.414228 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:31.414241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.416287 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.416293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:31.417384 (+ 1091us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.417422 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 06:56:31.417509 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 06:56:31.417634 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":909,"thread_start_us":939,"threads_started":1}
I20260621 06:56:31.418052 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.408350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.20.120.194:37783)
0621 06:56:31.408852 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:31.408858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:31.408877 (+ 19us) server_negotiation.cc:407] Connection header received
0621 06:56:31.408975 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:31.408982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:31.409209 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:31.409381 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:31.410190 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.411742 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.414373 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.415848 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.416000 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.416151 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:31.417565 (+ 1414us) server_negotiation.cc:299] Negotiation successful
0621 06:56:31.417726 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
I20260621 06:56:31.418843 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.406519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:46213 (local address 127.0.0.1:34056)
0621 06:56:31.407482 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:31.407526 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:31.407764 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:31.412308 (+ 4544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:31.412311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:31.412331 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:31.412703 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:31.412714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.414563 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.414568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:31.416551 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:31.416565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.418354 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.418358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:31.418468 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.418489 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 06:56:31.418536 (+ 47us) client_negotiation.cc:240] Negotiation successful
0621 06:56:31.418609 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":1,"thread_start_us":1753,"threads_started":1}
I20260621 06:56:31.419945 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.406812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.20.120.195:46213)
0621 06:56:31.411771 (+ 4959us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:31.411780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:31.411809 (+ 29us) server_negotiation.cc:407] Connection header received
0621 06:56:31.411892 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:31.411899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:31.412116 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:31.412862 (+ 746us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:31.413120 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.414451 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.416721 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.418152 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.419235 (+ 1083us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.419391 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:31.419450 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 06:56:31.419587 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4644,"thread_start_us":4482,"threads_started":1}
I20260621 06:56:31.420782 5754 tablet_service.cc:1467] Processing CreateTablet for tablet e16e6b6df42f4f8fb0dde86936984617 (DEFAULT_TABLE table=test-table [id=3c7dbbb79d5b4e48b6d25d7e7a3bcb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:31.421437 5888 tablet_service.cc:1467] Processing CreateTablet for tablet e16e6b6df42f4f8fb0dde86936984617 (DEFAULT_TABLE table=test-table [id=3c7dbbb79d5b4e48b6d25d7e7a3bcb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:31.422132 5754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e16e6b6df42f4f8fb0dde86936984617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:31.423067 5888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e16e6b6df42f4f8fb0dde86936984617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:31.423472 6022 tablet_service.cc:1467] Processing CreateTablet for tablet e16e6b6df42f4f8fb0dde86936984617 (DEFAULT_TABLE table=test-table [id=3c7dbbb79d5b4e48b6d25d7e7a3bcb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:31.424724 6022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e16e6b6df42f4f8fb0dde86936984617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:31.432668 6105 tablet_bootstrap.cc:492] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea: Bootstrap starting.
I20260621 06:56:31.435909 6105 tablet_bootstrap.cc:654] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:31.437502 6105 log.cc:824] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:31.447562 6105 tablet_bootstrap.cc:492] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea: No bootstrap required, opened a new log
I20260621 06:56:31.447810 6105 ts_tablet_manager.cc:1397] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260621 06:56:31.452670 6108 tablet_bootstrap.cc:492] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87: Bootstrap starting.
I20260621 06:56:31.453843 6106 tablet_bootstrap.cc:492] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2: Bootstrap starting.
I20260621 06:56:31.456389 6108 tablet_bootstrap.cc:654] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:31.457301 6106 tablet_bootstrap.cc:654] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:31.457594 6108 log.cc:824] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:31.458293 6105 raft_consensus.cc:348] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aca594064a749008c4ae5c96f038f87" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 46213 } } peers { permanent_uuid: "49843cd089fb42c796496110920209ea" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36625 } } peers { permanent_uuid: "c9201885dcab4257992a803668208ff2" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 37783 } }
I20260621 06:56:31.458701 6105 raft_consensus.cc:374] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:31.458819 6105 raft_consensus.cc:729] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49843cd089fb42c796496110920209ea, State: Initialized, Role: FOLLOWER
I20260621 06:56:31.459475 6105 consensus_queue.cc:260] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aca594064a749008c4ae5c96f038f87" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 46213 } } peers { permanent_uuid: "49843cd089fb42c796496110920209ea" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36625 } } peers { permanent_uuid: "c9201885dcab4257992a803668208ff2" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 37783 } }
I20260621 06:56:31.460150 6106 log.cc:824] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:31.463135 6105 ts_tablet_manager.cc:1428] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260621 06:56:31.466434 6108 tablet_bootstrap.cc:492] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87: No bootstrap required, opened a new log
I20260621 06:56:31.466701 6108 ts_tablet_manager.cc:1397] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260621 06:56:31.467140 6106 tablet_bootstrap.cc:492] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2: No bootstrap required, opened a new log
I20260621 06:56:31.467374 6106 ts_tablet_manager.cc:1397] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260621 06:56:31.474321 6108 raft_consensus.cc:348] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aca594064a749008c4ae5c96f038f87" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 46213 } } peers { permanent_uuid: "49843cd089fb42c796496110920209ea" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36625 } } peers { permanent_uuid: "c9201885dcab4257992a803668208ff2" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 37783 } }
I20260621 06:56:31.474778 6108 raft_consensus.cc:374] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:31.474884 6108 raft_consensus.cc:729] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aca594064a749008c4ae5c96f038f87, State: Initialized, Role: FOLLOWER
I20260621 06:56:31.474803 6106 raft_consensus.cc:348] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aca594064a749008c4ae5c96f038f87" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 46213 } } peers { permanent_uuid: "49843cd089fb42c796496110920209ea" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36625 } } peers { permanent_uuid: "c9201885dcab4257992a803668208ff2" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 37783 } }
I20260621 06:56:31.475169 6106 raft_consensus.cc:374] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:31.475265 6106 raft_consensus.cc:729] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9201885dcab4257992a803668208ff2, State: Initialized, Role: FOLLOWER
I20260621 06:56:31.475561 6108 consensus_queue.cc:260] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aca594064a749008c4ae5c96f038f87" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 46213 } } peers { permanent_uuid: "49843cd089fb42c796496110920209ea" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36625 } } peers { permanent_uuid: "c9201885dcab4257992a803668208ff2" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 37783 } }
I20260621 06:56:31.475760 6106 consensus_queue.cc:260] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aca594064a749008c4ae5c96f038f87" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 46213 } } peers { permanent_uuid: "49843cd089fb42c796496110920209ea" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36625 } } peers { permanent_uuid: "c9201885dcab4257992a803668208ff2" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 37783 } }
I20260621 06:56:31.476785 5953 heartbeater.cc:503] Master 127.20.120.254:41795 was elected leader, sending a full tablet report...
I20260621 06:56:31.476790 6087 heartbeater.cc:503] Master 127.20.120.254:41795 was elected leader, sending a full tablet report...
I20260621 06:56:31.477358 6108 ts_tablet_manager.cc:1428] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260621 06:56:31.477383 6106 ts_tablet_manager.cc:1428] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260621 06:56:31.510192 6112 raft_consensus.cc:482] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:56:31.510499 6112 raft_consensus.cc:504] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aca594064a749008c4ae5c96f038f87" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 46213 } } peers { permanent_uuid: "49843cd089fb42c796496110920209ea" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36625 } } peers { permanent_uuid: "c9201885dcab4257992a803668208ff2" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 37783 } }
I20260621 06:56:31.512137 6112 leader_election.cc:288] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49843cd089fb42c796496110920209ea (127.20.120.193:36625), c9201885dcab4257992a803668208ff2 (127.20.120.194:37783)
I20260621 06:56:31.517720 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.512523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:48665 (local address 127.20.120.194:37783)
0621 06:56:31.512790 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:31.512798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:31.513004 (+ 206us) server_negotiation.cc:407] Connection header received
0621 06:56:31.513116 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:31.513123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:31.513208 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:31.513312 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:31.513821 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.514560 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.516137 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.516877 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.517043 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.517112 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:31.517356 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 06:56:31.517475 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260621 06:56:31.517720 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.512513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:37783 (local address 127.20.120.195:48665)
0621 06:56:31.512775 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:31.512798 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:31.512932 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:31.513493 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:31.513498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:31.513529 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:31.513686 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:31.513693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.514711 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.514722 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:31.515969 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:31.515982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.517025 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.517030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:31.517205 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.517227 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 06:56:31.517333 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 06:56:31.517455 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260621 06:56:31.518580 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.512767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:33187 (local address 127.20.120.193:36625)
0621 06:56:31.512939 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:31.512943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:31.513744 (+ 801us) server_negotiation.cc:407] Connection header received
0621 06:56:31.513946 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:31.513951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:31.514026 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:31.514135 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:31.514625 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.515305 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.516924 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.517662 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.517829 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.517893 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:31.518225 (+ 332us) server_negotiation.cc:299] Negotiation successful
0621 06:56:31.518348 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 06:56:31.518687 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.512586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:36625 (local address 127.20.120.195:33187)
0621 06:56:31.513642 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:31.513656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:31.513793 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:31.514147 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:31.514151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:31.514173 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:31.514481 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:31.514488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.515496 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.515503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:31.516774 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:31.516783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.517838 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.517844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:31.518027 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.518055 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 06:56:31.518201 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 06:56:31.518316 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":109,"thread_start_us":412,"threads_started":1}
I20260621 06:56:31.518750 5908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16e6b6df42f4f8fb0dde86936984617" candidate_uuid: "8aca594064a749008c4ae5c96f038f87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9201885dcab4257992a803668208ff2" is_pre_election: true
I20260621 06:56:31.519307 5908 raft_consensus.cc:2454] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8aca594064a749008c4ae5c96f038f87 in term 0.
I20260621 06:56:31.519687 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16e6b6df42f4f8fb0dde86936984617" candidate_uuid: "8aca594064a749008c4ae5c96f038f87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49843cd089fb42c796496110920209ea" is_pre_election: true
I20260621 06:56:31.520002 5975 leader_election.cc:302] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8aca594064a749008c4ae5c96f038f87, c9201885dcab4257992a803668208ff2; no voters:
I20260621 06:56:31.520289 5774 raft_consensus.cc:2454] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8aca594064a749008c4ae5c96f038f87 in term 0.
I20260621 06:56:31.520399 6112 raft_consensus.cc:2781] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:56:31.520558 6112 raft_consensus.cc:482] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:56:31.520682 6112 raft_consensus.cc:3037] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:31.523068 6112 raft_consensus.cc:504] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aca594064a749008c4ae5c96f038f87" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 46213 } } peers { permanent_uuid: "49843cd089fb42c796496110920209ea" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36625 } } peers { permanent_uuid: "c9201885dcab4257992a803668208ff2" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 37783 } }
I20260621 06:56:31.523676 6112 leader_election.cc:288] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [CANDIDATE]: Term 1 election: Requested vote from peers 49843cd089fb42c796496110920209ea (127.20.120.193:36625), c9201885dcab4257992a803668208ff2 (127.20.120.194:37783)
I20260621 06:56:31.524047 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16e6b6df42f4f8fb0dde86936984617" candidate_uuid: "8aca594064a749008c4ae5c96f038f87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49843cd089fb42c796496110920209ea"
I20260621 06:56:31.524103 5908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16e6b6df42f4f8fb0dde86936984617" candidate_uuid: "8aca594064a749008c4ae5c96f038f87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9201885dcab4257992a803668208ff2"
I20260621 06:56:31.524241 5774 raft_consensus.cc:3037] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:31.524268 5908 raft_consensus.cc:3037] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:31.527458 5908 raft_consensus.cc:2454] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aca594064a749008c4ae5c96f038f87 in term 1.
I20260621 06:56:31.527457 5774 raft_consensus.cc:2454] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aca594064a749008c4ae5c96f038f87 in term 1.
I20260621 06:56:31.527806 5975 leader_election.cc:302] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8aca594064a749008c4ae5c96f038f87, c9201885dcab4257992a803668208ff2; no voters:
I20260621 06:56:31.528050 6112 raft_consensus.cc:2781] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:31.528487 6112 raft_consensus.cc:686] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 1 LEADER]: Becoming Leader. State: Replica: 8aca594064a749008c4ae5c96f038f87, State: Running, Role: LEADER
I20260621 06:56:31.528941 6112 consensus_queue.cc:237] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aca594064a749008c4ae5c96f038f87" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 46213 } } peers { permanent_uuid: "49843cd089fb42c796496110920209ea" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36625 } } peers { permanent_uuid: "c9201885dcab4257992a803668208ff2" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 37783 } }
I20260621 06:56:31.533532 5630 catalog_manager.cc:5495] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8aca594064a749008c4ae5c96f038f87 (127.20.120.195). New cstate: current_term: 1 leader_uuid: "8aca594064a749008c4ae5c96f038f87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aca594064a749008c4ae5c96f038f87" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 46213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49843cd089fb42c796496110920209ea" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9201885dcab4257992a803668208ff2" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 37783 } health_report { overall_health: UNKNOWN } } }
I20260621 06:56:31.561615 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.556885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34070 (local address 127.20.120.195:46213)
0621 06:56:31.557055 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:31.557059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:31.557075 (+ 16us) server_negotiation.cc:407] Connection header received
0621 06:56:31.557218 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:31.557221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:31.557285 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:31.557397 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:56:31.557697 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.558441 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.559725 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.560096 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.560182 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.560617 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:56:31.561094 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:56:31.561205 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:31.561265 (+ 60us) server_negotiation.cc:299] Negotiation successful
0621 06:56:31.561325 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 06:56:31.569856 5908 raft_consensus.cc:1261] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2 [term 1 FOLLOWER]: Refusing update from remote peer 8aca594064a749008c4ae5c96f038f87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:56:31.569857 5774 raft_consensus.cc:1261] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea [term 1 FOLLOWER]: Refusing update from remote peer 8aca594064a749008c4ae5c96f038f87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:56:31.570923 6115 consensus_queue.cc:1035] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49843cd089fb42c796496110920209ea" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 36625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:56:31.571295 6112 consensus_queue.cc:1035] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9201885dcab4257992a803668208ff2" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 37783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 06:56:31.582315 5820 tablet.cc:2369] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:56:31.585266 6118 mvcc.cc:205] Tried to move back new op lower bound from 7299174365458911232 to 7299174365306195968. Current Snapshot: MvccSnapshot[applied={T|T < 7299174365458911232}]
I20260621 06:56:31.608856 5629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:56:31.609138 5629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:56:31.612627 5629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b685a3fd5e14db596ea557979c9d092: Sending DeleteTablet for 3 replicas of tablet e16e6b6df42f4f8fb0dde86936984617
I20260621 06:56:31.613813 6022 tablet_service.cc:1514] Processing DeleteTablet for tablet e16e6b6df42f4f8fb0dde86936984617 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:31 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:34056
I20260621 06:56:31.613816 5754 tablet_service.cc:1514] Processing DeleteTablet for tablet e16e6b6df42f4f8fb0dde86936984617 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:31 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:37948
I20260621 06:56:31.614104 5888 tablet_service.cc:1514] Processing DeleteTablet for tablet e16e6b6df42f4f8fb0dde86936984617 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:31 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:41278
I20260621 06:56:31.615999 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 5691
I20260621 06:56:31.626816 6127 tablet_replica.cc:330] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87: stopping tablet replica
I20260621 06:56:31.627382 6127 raft_consensus.cc:2229] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:56:31.628283 6127 raft_consensus.cc:2258] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:31.629262 6129 tablet_replica.cc:330] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2: stopping tablet replica
I20260621 06:56:31.629679 6129 raft_consensus.cc:2229] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:56:31.630226 6129 raft_consensus.cc:2258] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:31.630674 6127 ts_tablet_manager.cc:1905] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:31.632161 6129 ts_tablet_manager.cc:1905] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:31.638052 6127 ts_tablet_manager.cc:1918] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:31.638244 6127 log.cc:1196] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/e16e6b6df42f4f8fb0dde86936984617
I20260621 06:56:31.638752 6127 ts_tablet_manager.cc:1939] T e16e6b6df42f4f8fb0dde86936984617 P 8aca594064a749008c4ae5c96f038f87: Deleting consensus metadata
I20260621 06:56:31.639036 6129 ts_tablet_manager.cc:1918] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:31.639186 6129 log.cc:1196] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/e16e6b6df42f4f8fb0dde86936984617
I20260621 06:56:31.639721 6129 ts_tablet_manager.cc:1939] T e16e6b6df42f4f8fb0dde86936984617 P c9201885dcab4257992a803668208ff2: Deleting consensus metadata
I20260621 06:56:31.640717 5617 catalog_manager.cc:4841] TS 8aca594064a749008c4ae5c96f038f87 (127.20.120.195:46213): tablet e16e6b6df42f4f8fb0dde86936984617 (table test-table [id=3c7dbbb79d5b4e48b6d25d7e7a3bcb3f]) successfully deleted
I20260621 06:56:31.641563 5614 catalog_manager.cc:4841] TS c9201885dcab4257992a803668208ff2 (127.20.120.194:37783): tablet e16e6b6df42f4f8fb0dde86936984617 (table test-table [id=3c7dbbb79d5b4e48b6d25d7e7a3bcb3f]) successfully deleted
I20260621 06:56:31.641825 6128 tablet_replica.cc:330] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea: stopping tablet replica
I20260621 06:56:31.642264 6128 raft_consensus.cc:2229] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:56:31.642709 6128 raft_consensus.cc:2258] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:31.644672 6128 ts_tablet_manager.cc:1905] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:31.650314 6128 ts_tablet_manager.cc:1918] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:31.650488 6128 log.cc:1196] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/e16e6b6df42f4f8fb0dde86936984617
I20260621 06:56:31.650915 6128 ts_tablet_manager.cc:1939] T e16e6b6df42f4f8fb0dde86936984617 P 49843cd089fb42c796496110920209ea: Deleting consensus metadata
Jun 21 06:56:31 dist-test-slave-gmf2 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:56:31.652681 5616 catalog_manager.cc:4841] TS 49843cd089fb42c796496110920209ea (127.20.120.193:36625): tablet e16e6b6df42f4f8fb0dde86936984617 (table test-table [id=3c7dbbb79d5b4e48b6d25d7e7a3bcb3f]) successfully deleted
I20260621 06:56:31.658728 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.617480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.20.120.193:36625)
0621 06:56:31.617657 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:31.617663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:31.640349 (+ 22686us) server_negotiation.cc:407] Connection header received
0621 06:56:31.641367 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:31.641376 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:31.641482 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:31.641565 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:31.643653 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.644781 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.646766 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.647432 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.647556 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.653065 (+ 5509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:31.653076 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:31.653095 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:31.653131 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:31.655887 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:31.656605 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:31.656612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:31.656614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:31.656742 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:31.657134 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:31.657138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:31.657140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:31.657469 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:31.657776 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:31.658135 (+ 359us) server_negotiation.cc:299] Negotiation successful
0621 06:56:31.658345 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 06:56:31.659353 5814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:31.785272 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 5691
I20260621 06:56:31.803855 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 5825
Jun 21 06:56:31 dist-test-slave-gmf2 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:56:31.823251 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.804713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.20.120.194:37783)
0621 06:56:31.804932 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:31.804938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:31.804960 (+ 22us) server_negotiation.cc:407] Connection header received
0621 06:56:31.805027 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:31.805033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:31.805129 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:31.805266 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:31.806568 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.807712 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.808976 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.809565 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.809693 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.816650 (+ 6957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:31.816671 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:31.816693 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:31.816746 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:31.820407 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:31.821153 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:31.821163 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:31.821167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:31.821316 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:31.821671 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:31.821685 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:31.821688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:31.822161 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:31.822460 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:31.822655 (+ 195us) server_negotiation.cc:299] Negotiation successful
0621 06:56:31.822868 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 06:56:31.824004 5948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:31.914412 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 5825
I20260621 06:56:31.935817 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 5959
Jun 21 06:56:31 dist-test-slave-gmf2 krb5kdc[5585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:56:31.959685 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:31.936925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.20.120.195:46213)
0621 06:56:31.937175 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:31.937211 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:31.937243 (+ 32us) server_negotiation.cc:407] Connection header received
0621 06:56:31.937343 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:31.937348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:31.937515 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:31.937699 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:31.939961 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.942052 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.944336 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:31.945162 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:31.945266 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:31.952070 (+ 6804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:31.952083 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:31.952100 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:31.952137 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:31.956230 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:31.957116 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:31.957130 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:31.957136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:31.957351 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:31.957807 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:31.957812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:31.957816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:31.958333 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:31.958660 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:31.958958 (+ 298us) server_negotiation.cc:299] Negotiation successful
0621 06:56:31.959196 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 06:56:31.960705 6082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:32.084808 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 5959
I20260621 06:56:32.104163 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 5598
I20260621 06:56:32.105216 5660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:32.214177 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 5598
2026-06-21T06:56:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4208 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-21T06:56:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:56:32Z Disabled control of system clock
I20260621 06:56:32.287076 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:42239
--webserver_interface=127.20.120.254
--webserver_port=0
--builtin_ntp_servers=127.20.120.212:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:42239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 06:56:32.599850 6138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:32.600112 6138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:32.600193 6138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:32.607192 6138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:56:32.607285 6138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:32.607321 6138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:32.607372 6138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:56:32.607414 6138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:56:32.616251 6138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:42239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:42239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:32.617491 6138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:32.619270 6138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:32.630764 6143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:32.630847 6146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:32.630820 6144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:32.631621 6138 server_base.cc:1034] running on GCE node
I20260621 06:56:32.632383 6138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:32.633694 6138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:32.634920 6138 hybrid_clock.cc:648] HybridClock initialized: now 1782024992634868 us; error 68 us; skew 500 ppm
I20260621 06:56:32.635251 6138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:56:32.637418 6138 webserver.cc:460] Webserver started at http://127.20.120.254:44653/ using document root <none> and password file <none>
I20260621 06:56:32.638156 6138 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:32.638278 6138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:32.638613 6138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:32.641377 6138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "d9ea3d65d11c48a6842c59e8e812036f"
format_stamp: "Formatted at 2026-06-21 06:56:32 on dist-test-slave-gmf2"
server_key: "a4c5bfe896d179e6d0078006a3884528"
server_key_iv: "ef9af4f022e527943547b33c277b4508"
server_key_version: "encryptionkey@0"
I20260621 06:56:32.642247 6138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "d9ea3d65d11c48a6842c59e8e812036f"
format_stamp: "Formatted at 2026-06-21 06:56:32 on dist-test-slave-gmf2"
server_key: "a4c5bfe896d179e6d0078006a3884528"
server_key_iv: "ef9af4f022e527943547b33c277b4508"
server_key_version: "encryptionkey@0"
I20260621 06:56:32.647190 6138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 06:56:32.650633 6152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:32.652006 6138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:56:32.652175 6138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "d9ea3d65d11c48a6842c59e8e812036f"
format_stamp: "Formatted at 2026-06-21 06:56:32 on dist-test-slave-gmf2"
server_key: "a4c5bfe896d179e6d0078006a3884528"
server_key_iv: "ef9af4f022e527943547b33c277b4508"
server_key_version: "encryptionkey@0"
I20260621 06:56:32.652383 6138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:32.675155 6138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:32.675985 6138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:32.676262 6138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:32.693186 6138 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:42239
I20260621 06:56:32.693204 6203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:42239 every 8 connection(s)
I20260621 06:56:32.694550 6138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:56:32.696980 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 6138
I20260621 06:56:32.697168 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:56:32.697608 20963 external_mini_cluster.cc:1428] Setting key 8eef95c2bcfb53ccfa2daa2c89a26f02
I20260621 06:56:32.698434 6204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:32.711969 6204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f: Bootstrap starting.
I20260621 06:56:32.715220 6204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:32.716455 6204 log.cc:824] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:32.716725 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:32.699472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.20.120.254:42239)
0621 06:56:32.700494 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:32.700507 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:32.700558 (+ 51us) server_negotiation.cc:407] Connection header received
0621 06:56:32.703063 (+ 2505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:32.703097 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:32.703659 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:32.704063 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:32.704584 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:32.706620 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:32.707903 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:32.708924 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:32.709116 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:32.713995 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:32.714006 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:32.714022 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:32.714718 (+ 696us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:32.714834 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:32.714860 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:32.714989 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:32.715190 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 06:56:32.715491 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":307,"threads_started":1}
I20260621 06:56:32.719321 6204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f: No bootstrap required, opened a new log
I20260621 06:56:32.726243 6204 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ea3d65d11c48a6842c59e8e812036f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42239 } }
I20260621 06:56:32.726591 6204 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:32.726693 6204 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ea3d65d11c48a6842c59e8e812036f, State: Initialized, Role: FOLLOWER
I20260621 06:56:32.727193 6204 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ea3d65d11c48a6842c59e8e812036f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42239 } }
I20260621 06:56:32.727430 6204 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:56:32.727572 6204 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:56:32.727726 6204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:32.729640 6204 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ea3d65d11c48a6842c59e8e812036f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42239 } }
I20260621 06:56:32.730151 6204 leader_election.cc:302] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9ea3d65d11c48a6842c59e8e812036f; no voters:
I20260621 06:56:32.730686 6204 leader_election.cc:288] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:56:32.730831 6209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:32.731609 6209 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [term 1 LEADER]: Becoming Leader. State: Replica: d9ea3d65d11c48a6842c59e8e812036f, State: Running, Role: LEADER
I20260621 06:56:32.732077 6209 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ea3d65d11c48a6842c59e8e812036f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42239 } }
I20260621 06:56:32.732546 6204 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:56:32.735273 6211 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9ea3d65d11c48a6842c59e8e812036f. Latest consensus state: current_term: 1 leader_uuid: "d9ea3d65d11c48a6842c59e8e812036f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ea3d65d11c48a6842c59e8e812036f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42239 } } }
I20260621 06:56:32.735311 6210 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9ea3d65d11c48a6842c59e8e812036f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ea3d65d11c48a6842c59e8e812036f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42239 } } }
I20260621 06:56:32.735649 6211 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:32.735699 6210 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:32.738972 6218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:56:32.743839 6218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:56:32.751788 6218 catalog_manager.cc:1347] Generated new cluster ID: 85f315f24b1841dea3bd5eac74057d01
I20260621 06:56:32.751890 6218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:56:32.773535 6218 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:56:32.775105 6218 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:56:32.781447 6218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f: Generated new TSK 0
I20260621 06:56:32.782207 6218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:56:32.787403 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:42239
--builtin_ntp_servers=127.20.120.212:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 06:56:33.113381 6228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:33.113691 6228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:33.113853 6228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:33.121031 6228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:33.121160 6228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:33.121340 6228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:56:33.130695 6228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:42239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:33.131966 6228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:33.133898 6228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:33.146822 6233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:33.146759 6234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:33.147156 6236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:33.147874 6228 server_base.cc:1034] running on GCE node
I20260621 06:56:33.148492 6228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:33.149385 6228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:33.150647 6228 hybrid_clock.cc:648] HybridClock initialized: now 1782024993150600 us; error 48 us; skew 500 ppm
I20260621 06:56:33.150957 6228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:56:33.153021 6228 webserver.cc:460] Webserver started at http://127.20.120.193:33473/ using document root <none> and password file <none>
I20260621 06:56:33.153680 6228 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:33.153810 6228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:33.154183 6228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:33.157078 6228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "3a98683f25fc4bd1a50139f3445dd79a"
format_stamp: "Formatted at 2026-06-21 06:56:33 on dist-test-slave-gmf2"
server_key: "9679f5cf71d7ae36329e441a885d0cb0"
server_key_iv: "9af79d73900e5224aebbab07e16dcf5e"
server_key_version: "encryptionkey@0"
I20260621 06:56:33.157826 6228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "3a98683f25fc4bd1a50139f3445dd79a"
format_stamp: "Formatted at 2026-06-21 06:56:33 on dist-test-slave-gmf2"
server_key: "9679f5cf71d7ae36329e441a885d0cb0"
server_key_iv: "9af79d73900e5224aebbab07e16dcf5e"
server_key_version: "encryptionkey@0"
I20260621 06:56:33.162580 6228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 06:56:33.165822 6242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:33.167147 6228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:56:33.167328 6228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "3a98683f25fc4bd1a50139f3445dd79a"
format_stamp: "Formatted at 2026-06-21 06:56:33 on dist-test-slave-gmf2"
server_key: "9679f5cf71d7ae36329e441a885d0cb0"
server_key_iv: "9af79d73900e5224aebbab07e16dcf5e"
server_key_version: "encryptionkey@0"
I20260621 06:56:33.167537 6228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:33.183503 6228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:33.184270 6228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:33.184564 6228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:33.185487 6228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:33.186887 6228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:33.187016 6228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:33.187148 6228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:33.187237 6228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:33.219987 6228 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:43211
I20260621 06:56:33.220031 6354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:43211 every 8 connection(s)
I20260621 06:56:33.221289 6228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:56:33.222664 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 6228
I20260621 06:56:33.222860 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:56:33.223392 20963 external_mini_cluster.cc:1428] Setting key bc53dfe55bfd841c18b46e30a277269a
I20260621 06:56:33.226472 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:42239
--builtin_ntp_servers=127.20.120.212:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 06:56:33.234086 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:33.224069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:57425 (local address 127.20.120.254:42239)
0621 06:56:33.224554 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:33.224561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:33.225893 (+ 1332us) server_negotiation.cc:407] Connection header received
0621 06:56:33.227739 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:33.227744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:33.227834 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:33.227952 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:33.228960 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:33.229660 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:33.231279 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:33.231687 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:33.231822 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:33.233205 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:33.233210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:33.233214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:33.233230 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:33.233281 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:33.233293 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:33.233372 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:33.233618 (+ 246us) server_negotiation.cc:299] Negotiation successful
0621 06:56:33.233693 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260621 06:56:33.234997 6358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:33.224719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:42239 (local address 127.20.120.193:57425)
0621 06:56:33.225745 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:33.225796 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:33.227494 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:33.228183 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:33.228191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:33.228273 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:33.228803 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:33.228818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:33.229846 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:33.229850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:33.231134 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:33.231145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:33.231905 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:33.231911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:33.232141 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:33.232841 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:56:33.232846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:33.232897 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:56:33.233002 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:33.233402 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:33.233414 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:33.233425 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 06:56:33.233618 (+ 193us) client_negotiation.cc:240] Negotiation successful
0621 06:56:33.233989 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":326,"threads_started":1}
I20260621 06:56:33.236545 6355 heartbeater.cc:347] Connected to a master server at 127.20.120.254:42239
I20260621 06:56:33.236840 6355 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:33.237705 6355 heartbeater.cc:511] Master 127.20.120.254:42239 requested a full tablet report, sending...
I20260621 06:56:33.239804 6169 ts_manager.cc:194] Registered new tserver with Master: 3a98683f25fc4bd1a50139f3445dd79a (127.20.120.193:43211)
I20260621 06:56:33.241237 6169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.193:57425
W20260621 06:56:33.551913 6359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:33.552165 6359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:33.552320 6359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:33.559413 6359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:33.559500 6359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:33.559679 6359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:56:33.568328 6359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:42239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:33.569598 6359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:33.571399 6359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:33.584394 6364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:33.584422 6365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:33.584470 6367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:33.585039 6359 server_base.cc:1034] running on GCE node
I20260621 06:56:33.585602 6359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:33.586597 6359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:33.587832 6359 hybrid_clock.cc:648] HybridClock initialized: now 1782024993587783 us; error 46 us; skew 500 ppm
I20260621 06:56:33.588138 6359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:56:33.590296 6359 webserver.cc:460] Webserver started at http://127.20.120.194:41459/ using document root <none> and password file <none>
I20260621 06:56:33.591012 6359 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:33.591145 6359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:33.591457 6359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:33.594201 6359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "7300980dce2f4e18944ae603d28e4c62"
format_stamp: "Formatted at 2026-06-21 06:56:33 on dist-test-slave-gmf2"
server_key: "34dab7f3f833daec914d5a10471a902c"
server_key_iv: "973999f56349d8b68360604c93536007"
server_key_version: "encryptionkey@0"
I20260621 06:56:33.594960 6359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "7300980dce2f4e18944ae603d28e4c62"
format_stamp: "Formatted at 2026-06-21 06:56:33 on dist-test-slave-gmf2"
server_key: "34dab7f3f833daec914d5a10471a902c"
server_key_iv: "973999f56349d8b68360604c93536007"
server_key_version: "encryptionkey@0"
I20260621 06:56:33.600018 6359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260621 06:56:33.603434 6373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:33.604642 6359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 06:56:33.604820 6359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "7300980dce2f4e18944ae603d28e4c62"
format_stamp: "Formatted at 2026-06-21 06:56:33 on dist-test-slave-gmf2"
server_key: "34dab7f3f833daec914d5a10471a902c"
server_key_iv: "973999f56349d8b68360604c93536007"
server_key_version: "encryptionkey@0"
I20260621 06:56:33.605027 6359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:33.632045 6359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:33.632897 6359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:33.633181 6359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:33.634195 6359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:33.635592 6359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:33.635710 6359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:33.635845 6359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:33.635936 6359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:33.668087 6359 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:33337
I20260621 06:56:33.668139 6485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:33337 every 8 connection(s)
I20260621 06:56:33.669337 6359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:56:33.672505 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 6359
I20260621 06:56:33.672648 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:56:33.673007 20963 external_mini_cluster.cc:1428] Setting key 1ef09dd9d219f0c6bb67703a6d30ba06
I20260621 06:56:33.675817 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:42239
--builtin_ntp_servers=127.20.120.212:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 06:56:33.682790 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:33.672021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:58305 (local address 127.20.120.254:42239)
0621 06:56:33.672214 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:33.672221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:33.673779 (+ 1558us) server_negotiation.cc:407] Connection header received
0621 06:56:33.675706 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:33.675711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:33.675790 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:33.675904 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:33.676996 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:33.678181 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:33.679650 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:33.680435 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:33.680585 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:33.681862 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:33.681871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:33.681878 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:33.681902 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:33.681965 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:33.681983 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:33.682076 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:33.682291 (+ 215us) server_negotiation.cc:299] Negotiation successful
0621 06:56:33.682401 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 06:56:33.683568 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:33.672535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:42239 (local address 127.20.120.194:58305)
0621 06:56:33.673633 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:33.673674 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:33.675461 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:33.676214 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:33.676222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:33.676304 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:33.676835 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:33.676851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:33.678324 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:33.678327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:33.679518 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:33.679529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:33.680656 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:33.680661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:33.680862 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:33.681562 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:56:33.681565 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:33.681600 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:56:33.681682 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:33.682090 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:33.682098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:33.682104 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 06:56:33.682250 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 06:56:33.682525 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":338,"threads_started":1}
I20260621 06:56:33.684913 6486 heartbeater.cc:347] Connected to a master server at 127.20.120.254:42239
I20260621 06:56:33.685199 6486 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:33.685863 6486 heartbeater.cc:511] Master 127.20.120.254:42239 requested a full tablet report, sending...
I20260621 06:56:33.687206 6169 ts_manager.cc:194] Registered new tserver with Master: 7300980dce2f4e18944ae603d28e4c62 (127.20.120.194:33337)
I20260621 06:56:33.687929 6169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.194:58305
W20260621 06:56:33.998406 6489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:33.998716 6489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:33.998895 6489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:34.006145 6489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:34.006278 6489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:34.006462 6489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:56:34.015242 6489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:42239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:34.016501 6489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:34.018388 6489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:34.031553 6495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:34.031586 6497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:34.031613 6494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:34.032318 6489 server_base.cc:1034] running on GCE node
I20260621 06:56:34.032918 6489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:34.033985 6489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:34.035220 6489 hybrid_clock.cc:648] HybridClock initialized: now 1782024994035174 us; error 56 us; skew 500 ppm
I20260621 06:56:34.035573 6489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:56:34.037801 6489 webserver.cc:460] Webserver started at http://127.20.120.195:42535/ using document root <none> and password file <none>
I20260621 06:56:34.038477 6489 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:34.038614 6489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:34.038913 6489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:34.041744 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "80dce2842df64ee8bdec81de6de28a9a"
format_stamp: "Formatted at 2026-06-21 06:56:34 on dist-test-slave-gmf2"
server_key: "616eca71acff97c6adaddc8468c59feb"
server_key_iv: "ff047a1ccc6bed10c8f594a95e211d33"
server_key_version: "encryptionkey@0"
I20260621 06:56:34.042569 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "80dce2842df64ee8bdec81de6de28a9a"
format_stamp: "Formatted at 2026-06-21 06:56:34 on dist-test-slave-gmf2"
server_key: "616eca71acff97c6adaddc8468c59feb"
server_key_iv: "ff047a1ccc6bed10c8f594a95e211d33"
server_key_version: "encryptionkey@0"
I20260621 06:56:34.047427 6489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260621 06:56:34.051100 6503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:34.052403 6489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:56:34.052575 6489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "80dce2842df64ee8bdec81de6de28a9a"
format_stamp: "Formatted at 2026-06-21 06:56:34 on dist-test-slave-gmf2"
server_key: "616eca71acff97c6adaddc8468c59feb"
server_key_iv: "ff047a1ccc6bed10c8f594a95e211d33"
server_key_version: "encryptionkey@0"
I20260621 06:56:34.052799 6489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:34.072788 6489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:34.073565 6489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:34.073868 6489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:34.074813 6489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:34.076267 6489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:34.076398 6489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:34.076519 6489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:34.076606 6489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:34.109993 6489 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:36665
I20260621 06:56:34.110105 6615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:36665 every 8 connection(s)
I20260621 06:56:34.111291 6489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:56:34.121058 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 6489
I20260621 06:56:34.121260 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:56:34.121698 20963 external_mini_cluster.cc:1428] Setting key 4b44e05b86d5bdec8787f6ae42efb5c1
I20260621 06:56:34.123907 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.113867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:52367 (local address 127.20.120.254:42239)
0621 06:56:34.114071 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:34.114075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:34.115440 (+ 1365us) server_negotiation.cc:407] Connection header received
0621 06:56:34.117206 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:34.117213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:34.117307 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:34.117420 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:34.118632 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.119373 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.120910 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.121415 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.121556 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.123077 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:34.123083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:34.123088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:34.123112 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:34.123151 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:34.123167 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:34.123263 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:34.123519 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 06:56:34.123615 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 06:56:34.125026 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.114436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:42239 (local address 127.20.120.195:52367)
0621 06:56:34.115291 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.115325 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:34.116931 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:34.117656 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:34.117663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:34.117751 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:34.118449 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:34.118467 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.119530 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.119541 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:34.120771 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:34.120783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.121641 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.121650 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:34.121965 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.122721 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:56:34.122724 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:34.122762 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:56:34.122836 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:34.123262 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:34.123270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:34.123275 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 06:56:34.123468 (+ 193us) client_negotiation.cc:240] Negotiation successful
0621 06:56:34.123839 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":263,"threads_started":1}
I20260621 06:56:34.126268 6616 heartbeater.cc:347] Connected to a master server at 127.20.120.254:42239
I20260621 06:56:34.126518 6616 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:34.127177 6616 heartbeater.cc:511] Master 127.20.120.254:42239 requested a full tablet report, sending...
I20260621 06:56:34.128386 6169 ts_manager.cc:194] Registered new tserver with Master: 80dce2842df64ee8bdec81de6de28a9a (127.20.120.195:36665)
I20260621 06:56:34.129120 6169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.195:52367
I20260621 06:56:34.136727 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:56:34.146534 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.140716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.20.120.254:42239)
0621 06:56:34.140956 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:34.140967 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:34.141330 (+ 363us) server_negotiation.cc:407] Connection header received
0621 06:56:34.141458 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:34.141462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:34.141542 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:34.141648 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:34.142162 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.142904 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.144267 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.144660 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.144754 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.145836 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:34.145840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:34.145843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:34.145858 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:34.145885 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:34.145896 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:34.146032 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:34.146133 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 06:56:34.146202 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 06:56:34.150640 6169 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:56:34.157631 6169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:56:34.190619 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.168802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43211 (local address 127.0.0.1:35984)
0621 06:56:34.178511 (+ 9709us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.178556 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:34.178727 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:34.182839 (+ 4112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:34.182850 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:34.182907 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:34.183442 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:34.183460 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.186021 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.186031 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:34.187624 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:34.187641 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.189594 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.189602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:34.189883 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.189970 (+ 87us) client_negotiation.cc:769] Sending connection context
0621 06:56:34.190109 (+ 139us) client_negotiation.cc:240] Negotiation successful
0621 06:56:34.190282 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9479,"thread_start_us":17288,"threads_started":1}
I20260621 06:56:34.190666 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.169036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.20.120.193:43211)
0621 06:56:34.181829 (+ 12793us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:34.181842 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:34.181890 (+ 48us) server_negotiation.cc:407] Connection header received
0621 06:56:34.182025 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:34.182036 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:34.182318 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:34.183704 (+ 1386us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:34.184081 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.185789 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.187818 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.189341 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.189528 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.189776 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:34.190144 (+ 368us) server_negotiation.cc:299] Negotiation successful
0621 06:56:34.190317 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12557,"thread_start_us":13436,"threads_started":1}
I20260621 06:56:34.194504 6290 tablet_service.cc:1467] Processing CreateTablet for tablet 51ae86e65f6648dc87c2fc40abbb5b5f (DEFAULT_TABLE table=test-table [id=3483f8ba21d34c6ab0d4c772aeb4eb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:34.195991 6290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ae86e65f6648dc87c2fc40abbb5b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:34.197538 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.170744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.20.120.194:33337)
0621 06:56:34.184905 (+ 14161us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:34.184917 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:34.188232 (+ 3315us) server_negotiation.cc:407] Connection header received
0621 06:56:34.188344 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:34.188350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:34.188585 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:34.188770 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:34.189607 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.191256 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.194781 (+ 3525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.196009 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.196173 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.196408 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:34.197024 (+ 616us) server_negotiation.cc:299] Negotiation successful
0621 06:56:34.197197 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13951,"thread_start_us":14166,"threads_started":1}
I20260621 06:56:34.197533 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.170022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:33337 (local address 127.0.0.1:59596)
0621 06:56:34.187965 (+ 17943us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.187983 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:34.188101 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:34.189008 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:34.189013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:34.189042 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:34.189405 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:34.189439 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.191472 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.191478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:34.194505 (+ 3027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:34.194517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.196540 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.196548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:34.196763 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.196846 (+ 83us) client_negotiation.cc:769] Sending connection context
0621 06:56:34.196965 (+ 119us) client_negotiation.cc:240] Negotiation successful
0621 06:56:34.197129 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17842,"mutex_wait_us":175,"thread_start_us":8126,"threads_started":1}
I20260621 06:56:34.197623 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.169876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60906 (local address 127.20.120.195:36665)
0621 06:56:34.188442 (+ 18566us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:34.188450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:34.188478 (+ 28us) server_negotiation.cc:407] Connection header received
0621 06:56:34.188575 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:34.188582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:34.188785 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:34.188962 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:34.190106 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.191632 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.194723 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.196105 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.196290 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.196488 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:34.197117 (+ 629us) server_negotiation.cc:299] Negotiation successful
0621 06:56:34.197278 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18406,"thread_start_us":19065,"threads_started":1}
I20260621 06:56:34.198215 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.169152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:36665 (local address 127.0.0.1:60906)
0621 06:56:34.185646 (+ 16494us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.185669 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:34.185817 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:34.189240 (+ 3423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:34.189246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:34.189284 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:34.189887 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:34.189898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.191826 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.191832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:34.194478 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:34.194500 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.196614 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.196624 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:34.196850 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.196880 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 06:56:34.196961 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 06:56:34.197885 (+ 924us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16305,"mutex_wait_us":990,"thread_start_us":16114,"threads_started":1}
I20260621 06:56:34.201807 6421 tablet_service.cc:1467] Processing CreateTablet for tablet 51ae86e65f6648dc87c2fc40abbb5b5f (DEFAULT_TABLE table=test-table [id=3483f8ba21d34c6ab0d4c772aeb4eb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:34.201805 6551 tablet_service.cc:1467] Processing CreateTablet for tablet 51ae86e65f6648dc87c2fc40abbb5b5f (DEFAULT_TABLE table=test-table [id=3483f8ba21d34c6ab0d4c772aeb4eb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:34.203337 6421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ae86e65f6648dc87c2fc40abbb5b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:34.203336 6551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ae86e65f6648dc87c2fc40abbb5b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:34.208190 6634 tablet_bootstrap.cc:492] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a: Bootstrap starting.
I20260621 06:56:34.211265 6634 tablet_bootstrap.cc:654] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:34.212273 6634 log.cc:824] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:34.215250 6634 tablet_bootstrap.cc:492] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a: No bootstrap required, opened a new log
I20260621 06:56:34.215535 6634 ts_tablet_manager.cc:1397] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260621 06:56:34.218201 6636 tablet_bootstrap.cc:492] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62: Bootstrap starting.
I20260621 06:56:34.218887 6637 tablet_bootstrap.cc:492] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a: Bootstrap starting.
I20260621 06:56:34.222381 6637 tablet_bootstrap.cc:654] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:34.223613 6637 log.cc:824] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:34.225098 6636 tablet_bootstrap.cc:654] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:34.225355 6634 raft_consensus.cc:348] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a98683f25fc4bd1a50139f3445dd79a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43211 } } peers { permanent_uuid: "80dce2842df64ee8bdec81de6de28a9a" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 36665 } } peers { permanent_uuid: "7300980dce2f4e18944ae603d28e4c62" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33337 } }
I20260621 06:56:34.225766 6634 raft_consensus.cc:374] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:34.225857 6634 raft_consensus.cc:729] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a98683f25fc4bd1a50139f3445dd79a, State: Initialized, Role: FOLLOWER
I20260621 06:56:34.226559 6634 consensus_queue.cc:260] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a98683f25fc4bd1a50139f3445dd79a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43211 } } peers { permanent_uuid: "80dce2842df64ee8bdec81de6de28a9a" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 36665 } } peers { permanent_uuid: "7300980dce2f4e18944ae603d28e4c62" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33337 } }
I20260621 06:56:34.227102 6636 log.cc:824] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:34.227258 6637 tablet_bootstrap.cc:492] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a: No bootstrap required, opened a new log
I20260621 06:56:34.227556 6637 ts_tablet_manager.cc:1397] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260621 06:56:34.236213 6636 tablet_bootstrap.cc:492] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62: No bootstrap required, opened a new log
I20260621 06:56:34.236521 6636 ts_tablet_manager.cc:1397] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.008s
I20260621 06:56:34.241389 6637 raft_consensus.cc:348] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a98683f25fc4bd1a50139f3445dd79a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43211 } } peers { permanent_uuid: "80dce2842df64ee8bdec81de6de28a9a" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 36665 } } peers { permanent_uuid: "7300980dce2f4e18944ae603d28e4c62" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33337 } }
I20260621 06:56:34.241966 6637 raft_consensus.cc:374] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:34.242125 6637 raft_consensus.cc:729] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80dce2842df64ee8bdec81de6de28a9a, State: Initialized, Role: FOLLOWER
I20260621 06:56:34.242873 6637 consensus_queue.cc:260] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a98683f25fc4bd1a50139f3445dd79a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43211 } } peers { permanent_uuid: "80dce2842df64ee8bdec81de6de28a9a" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 36665 } } peers { permanent_uuid: "7300980dce2f4e18944ae603d28e4c62" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33337 } }
I20260621 06:56:34.244745 6355 heartbeater.cc:503] Master 127.20.120.254:42239 was elected leader, sending a full tablet report...
I20260621 06:56:34.246655 6634 ts_tablet_manager.cc:1428] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a: Time spent starting tablet: real 0.031s user 0.016s sys 0.016s
I20260621 06:56:34.247561 6616 heartbeater.cc:503] Master 127.20.120.254:42239 was elected leader, sending a full tablet report...
I20260621 06:56:34.248119 6637 ts_tablet_manager.cc:1428] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a: Time spent starting tablet: real 0.020s user 0.006s sys 0.011s
I20260621 06:56:34.249653 6636 raft_consensus.cc:348] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a98683f25fc4bd1a50139f3445dd79a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43211 } } peers { permanent_uuid: "80dce2842df64ee8bdec81de6de28a9a" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 36665 } } peers { permanent_uuid: "7300980dce2f4e18944ae603d28e4c62" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33337 } }
I20260621 06:56:34.250075 6636 raft_consensus.cc:374] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:34.250198 6636 raft_consensus.cc:729] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7300980dce2f4e18944ae603d28e4c62, State: Initialized, Role: FOLLOWER
I20260621 06:56:34.250874 6636 consensus_queue.cc:260] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a98683f25fc4bd1a50139f3445dd79a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43211 } } peers { permanent_uuid: "80dce2842df64ee8bdec81de6de28a9a" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 36665 } } peers { permanent_uuid: "7300980dce2f4e18944ae603d28e4c62" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33337 } }
I20260621 06:56:34.251964 6486 heartbeater.cc:503] Master 127.20.120.254:42239 was elected leader, sending a full tablet report...
I20260621 06:56:34.253407 6636 ts_tablet_manager.cc:1428] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62: Time spent starting tablet: real 0.017s user 0.007s sys 0.006s
W20260621 06:56:34.363031 6617 tablet.cc:2369] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:56:34.421765 6487 tablet.cc:2369] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:56:34.467478 6639 raft_consensus.cc:482] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:56:34.467760 6639 raft_consensus.cc:504] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a98683f25fc4bd1a50139f3445dd79a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43211 } } peers { permanent_uuid: "80dce2842df64ee8bdec81de6de28a9a" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 36665 } } peers { permanent_uuid: "7300980dce2f4e18944ae603d28e4c62" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33337 } }
I20260621 06:56:34.469388 6639 leader_election.cc:288] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80dce2842df64ee8bdec81de6de28a9a (127.20.120.195:36665), 7300980dce2f4e18944ae603d28e4c62 (127.20.120.194:33337)
W20260621 06:56:34.473734 6356 tablet.cc:2369] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:56:34.476883 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.469880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:46155 (local address 127.20.120.194:33337)
0621 06:56:34.470322 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:34.470332 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:34.470940 (+ 608us) server_negotiation.cc:407] Connection header received
0621 06:56:34.471158 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:34.471162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:34.471227 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:34.471324 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:34.472145 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.472829 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.475003 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.475998 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.476119 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.476166 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:34.476503 (+ 337us) server_negotiation.cc:299] Negotiation successful
0621 06:56:34.476628 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260621 06:56:34.476907 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.470024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:33337 (local address 127.20.120.193:46155)
0621 06:56:34.470853 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.470867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:34.470965 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:34.471341 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:34.471345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:34.471384 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:34.471962 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:34.471973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.472954 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.472961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:34.474808 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:34.474822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.476169 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.476176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:34.476345 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.476378 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 06:56:34.476465 (+ 87us) client_negotiation.cc:240] Negotiation successful
0621 06:56:34.476591 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":82,"thread_start_us":565,"threads_started":1}
I20260621 06:56:34.477317 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.469798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:36665 (local address 127.20.120.193:38589)
0621 06:56:34.470640 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.470666 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:34.470845 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:34.471230 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:34.471235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:34.471261 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:34.471901 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:34.471918 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.472953 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.472959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:34.475050 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:34.475065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.476567 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.476574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:34.476717 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.476754 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 06:56:34.476858 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 06:56:34.477002 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":65,"thread_start_us":542,"threads_started":1}
I20260621 06:56:34.477407 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.469759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:38589 (local address 127.20.120.195:36665)
0621 06:56:34.470007 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:34.470015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:34.470778 (+ 763us) server_negotiation.cc:407] Connection header received
0621 06:56:34.471023 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:34.471028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:34.471100 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:34.471212 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:34.472084 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.472814 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.475236 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.476351 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.476510 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.476584 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:34.476903 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 06:56:34.477181 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 06:56:34.478130 6441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ae86e65f6648dc87c2fc40abbb5b5f" candidate_uuid: "3a98683f25fc4bd1a50139f3445dd79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7300980dce2f4e18944ae603d28e4c62" is_pre_election: true
I20260621 06:56:34.478420 6571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ae86e65f6648dc87c2fc40abbb5b5f" candidate_uuid: "3a98683f25fc4bd1a50139f3445dd79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80dce2842df64ee8bdec81de6de28a9a" is_pre_election: true
I20260621 06:56:34.478627 6441 raft_consensus.cc:2454] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a98683f25fc4bd1a50139f3445dd79a in term 0.
I20260621 06:56:34.478916 6571 raft_consensus.cc:2454] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a98683f25fc4bd1a50139f3445dd79a in term 0.
I20260621 06:56:34.479269 6244 leader_election.cc:302] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a98683f25fc4bd1a50139f3445dd79a, 7300980dce2f4e18944ae603d28e4c62; no voters:
I20260621 06:56:34.479693 6639 raft_consensus.cc:2781] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:56:34.479839 6639 raft_consensus.cc:482] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:56:34.479940 6639 raft_consensus.cc:3037] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:34.482154 6639 raft_consensus.cc:504] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a98683f25fc4bd1a50139f3445dd79a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43211 } } peers { permanent_uuid: "80dce2842df64ee8bdec81de6de28a9a" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 36665 } } peers { permanent_uuid: "7300980dce2f4e18944ae603d28e4c62" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33337 } }
I20260621 06:56:34.482728 6639 leader_election.cc:288] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [CANDIDATE]: Term 1 election: Requested vote from peers 80dce2842df64ee8bdec81de6de28a9a (127.20.120.195:36665), 7300980dce2f4e18944ae603d28e4c62 (127.20.120.194:33337)
I20260621 06:56:34.483163 6441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ae86e65f6648dc87c2fc40abbb5b5f" candidate_uuid: "3a98683f25fc4bd1a50139f3445dd79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7300980dce2f4e18944ae603d28e4c62"
I20260621 06:56:34.483173 6571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ae86e65f6648dc87c2fc40abbb5b5f" candidate_uuid: "3a98683f25fc4bd1a50139f3445dd79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80dce2842df64ee8bdec81de6de28a9a"
I20260621 06:56:34.483345 6441 raft_consensus.cc:3037] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:34.483352 6571 raft_consensus.cc:3037] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:34.485291 6571 raft_consensus.cc:2454] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a98683f25fc4bd1a50139f3445dd79a in term 1.
I20260621 06:56:34.485320 6441 raft_consensus.cc:2454] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a98683f25fc4bd1a50139f3445dd79a in term 1.
I20260621 06:56:34.485658 6245 leader_election.cc:302] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a98683f25fc4bd1a50139f3445dd79a, 80dce2842df64ee8bdec81de6de28a9a; no voters:
I20260621 06:56:34.485980 6639 raft_consensus.cc:2781] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:34.486393 6639 raft_consensus.cc:686] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 1 LEADER]: Becoming Leader. State: Replica: 3a98683f25fc4bd1a50139f3445dd79a, State: Running, Role: LEADER
I20260621 06:56:34.486812 6639 consensus_queue.cc:237] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a98683f25fc4bd1a50139f3445dd79a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43211 } } peers { permanent_uuid: "80dce2842df64ee8bdec81de6de28a9a" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 36665 } } peers { permanent_uuid: "7300980dce2f4e18944ae603d28e4c62" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33337 } }
I20260621 06:56:34.491236 6169 catalog_manager.cc:5495] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a98683f25fc4bd1a50139f3445dd79a (127.20.120.193). New cstate: current_term: 1 leader_uuid: "3a98683f25fc4bd1a50139f3445dd79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a98683f25fc4bd1a50139f3445dd79a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80dce2842df64ee8bdec81de6de28a9a" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 36665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7300980dce2f4e18944ae603d28e4c62" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33337 } health_report { overall_health: UNKNOWN } } }
I20260621 06:56:34.558531 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.552950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.20.120.193:43211)
0621 06:56:34.553146 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:34.553150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:34.553168 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:56:34.553261 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:34.553265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:34.553365 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:34.553513 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:56:34.553853 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.554666 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.556365 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.556810 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.556967 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.557441 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:56:34.557955 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:56:34.558086 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:34.558160 (+ 74us) server_negotiation.cc:299] Negotiation successful
0621 06:56:34.558221 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 06:56:34.566639 6571 raft_consensus.cc:1261] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a [term 1 FOLLOWER]: Refusing update from remote peer 3a98683f25fc4bd1a50139f3445dd79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:56:34.566643 6441 raft_consensus.cc:1261] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62 [term 1 FOLLOWER]: Refusing update from remote peer 3a98683f25fc4bd1a50139f3445dd79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:56:34.567647 6645 consensus_queue.cc:1035] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80dce2842df64ee8bdec81de6de28a9a" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 36665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:56:34.567965 6639 consensus_queue.cc:1035] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7300980dce2f4e18944ae603d28e4c62" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 33337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:56:34.579738 6648 mvcc.cc:205] Tried to move back new op lower bound from 7299174377734889472 to 7299174377421553664. Current Snapshot: MvccSnapshot[applied={T|T < 7299174377734889472}]
I20260621 06:56:34.591166 6168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:56:34.591473 6168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:56:34.595049 6168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9ea3d65d11c48a6842c59e8e812036f: Sending DeleteTablet for 3 replicas of tablet 51ae86e65f6648dc87c2fc40abbb5b5f
I20260621 06:56:34.596024 6290 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ae86e65f6648dc87c2fc40abbb5b5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:34 UTC) from {username='slave'} at 127.0.0.1:35984
I20260621 06:56:34.596146 6551 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ae86e65f6648dc87c2fc40abbb5b5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:34 UTC) from {username='slave'} at 127.0.0.1:60906
I20260621 06:56:34.596186 6421 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ae86e65f6648dc87c2fc40abbb5b5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:34 UTC) from {username='slave'} at 127.0.0.1:59596
I20260621 06:56:34.597186 6659 tablet_replica.cc:330] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a: stopping tablet replica
I20260621 06:56:34.597604 6659 raft_consensus.cc:2229] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:56:34.597679 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 6228
I20260621 06:56:34.598120 6659 raft_consensus.cc:2258] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:34.598845 6658 tablet_replica.cc:330] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62: stopping tablet replica
I20260621 06:56:34.599239 6658 raft_consensus.cc:2229] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:56:34.599697 6658 raft_consensus.cc:2258] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:34.601478 6659 ts_tablet_manager.cc:1905] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:34.601496 6658 ts_tablet_manager.cc:1905] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:34.608588 6659 ts_tablet_manager.cc:1918] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:34.608788 6659 log.cc:1196] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/51ae86e65f6648dc87c2fc40abbb5b5f
I20260621 06:56:34.609469 6659 ts_tablet_manager.cc:1939] T 51ae86e65f6648dc87c2fc40abbb5b5f P 80dce2842df64ee8bdec81de6de28a9a: Deleting consensus metadata
I20260621 06:56:34.609911 6658 ts_tablet_manager.cc:1918] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:34.610141 6658 log.cc:1196] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/51ae86e65f6648dc87c2fc40abbb5b5f
I20260621 06:56:34.610739 6658 ts_tablet_manager.cc:1939] T 51ae86e65f6648dc87c2fc40abbb5b5f P 7300980dce2f4e18944ae603d28e4c62: Deleting consensus metadata
I20260621 06:56:34.611383 6155 catalog_manager.cc:4841] TS 80dce2842df64ee8bdec81de6de28a9a (127.20.120.195:36665): tablet 51ae86e65f6648dc87c2fc40abbb5b5f (table test-table [id=3483f8ba21d34c6ab0d4c772aeb4eb55]) successfully deleted
I20260621 06:56:34.612424 6154 catalog_manager.cc:4841] TS 7300980dce2f4e18944ae603d28e4c62 (127.20.120.194:33337): tablet 51ae86e65f6648dc87c2fc40abbb5b5f (table test-table [id=3483f8ba21d34c6ab0d4c772aeb4eb55]) successfully deleted
I20260621 06:56:34.612514 6657 tablet_replica.cc:330] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a: stopping tablet replica
I20260621 06:56:34.612936 6657 raft_consensus.cc:2229] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:56:34.613637 6657 raft_consensus.cc:2258] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:34.615743 6657 ts_tablet_manager.cc:1905] T 51ae86e65f6648dc87c2fc40abbb5b5f P 3a98683f25fc4bd1a50139f3445dd79a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:34.618407 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.600026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.20.120.193:43211)
0621 06:56:34.600473 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:34.600479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:34.611825 (+ 11346us) server_negotiation.cc:407] Connection header received
0621 06:56:34.612059 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:34.612068 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:34.612177 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:34.612285 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:34.613048 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.614014 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.615308 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.615766 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.615917 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.616995 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:34.617003 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:34.617018 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:34.617576 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:34.617636 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:34.617654 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:34.617772 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:34.617957 (+ 185us) server_negotiation.cc:299] Negotiation successful
0621 06:56:34.618058 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260621 06:56:34.619230 6350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:34.726110 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 6228
I20260621 06:56:34.744814 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 6359
W20260621 06:56:34.745002 6156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.120.193:43211 (error 108)
I20260621 06:56:34.746942 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.746209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43211 (local address 127.0.0.1:36014)
0621 06:56:34.746624 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.746745 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":194,"threads_started":1}
W20260621 06:56:34.747179 6156 catalog_manager.cc:4584] TS 3a98683f25fc4bd1a50139f3445dd79a (127.20.120.193:43211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51ae86e65f6648dc87c2fc40abbb5b5f: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43211: connect: Connection refused (error 111)
I20260621 06:56:34.751653 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.745611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.20.120.194:33337)
0621 06:56:34.746173 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:34.746178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:34.746193 (+ 15us) server_negotiation.cc:407] Connection header received
0621 06:56:34.746244 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:34.746248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:34.746385 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:34.746500 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:34.746925 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.747784 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.748959 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.749330 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.749418 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.750365 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:34.750373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:34.750386 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:34.750948 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:34.750995 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:34.751013 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:34.751105 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:34.751213 (+ 108us) server_negotiation.cc:299] Negotiation successful
0621 06:56:34.751293 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381}
I20260621 06:56:34.752228 6481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:34.800922 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.800364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43211 (local address 127.0.0.1:36018)
0621 06:56:34.800630 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.800742 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260621 06:56:34.802152 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.801701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43211 (local address 127.0.0.1:36034)
0621 06:56:34.801865 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.801995 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260621 06:56:34.839713 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 6359
I20260621 06:56:34.856688 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 6489
I20260621 06:56:34.862567 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.862175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43211 (local address 127.0.0.1:36044)
0621 06:56:34.862353 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.862426 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260621 06:56:34.863076 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.857478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.20.120.195:36665)
0621 06:56:34.857657 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:34.857661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:34.857679 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:56:34.857810 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:34.857814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:34.857977 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:34.858063 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:34.858421 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.859123 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.860248 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:34.860621 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:34.860711 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:34.861648 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:34.861658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:34.861673 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:34.862314 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:34.862359 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:34.862378 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:34.862505 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:34.862640 (+ 135us) server_negotiation.cc:299] Negotiation successful
0621 06:56:34.862745 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 06:56:34.863698 6611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:34.863791 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.863342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43211 (local address 127.0.0.1:36046)
0621 06:56:34.863520 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.863615 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260621 06:56:34.944473 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.943912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43211 (local address 127.0.0.1:36060)
0621 06:56:34.944231 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.944330 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260621 06:56:34.945508 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:34.945134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43211 (local address 127.0.0.1:36072)
0621 06:56:34.945318 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:34.945390 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260621 06:56:34.951848 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 6489
I20260621 06:56:34.968530 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 6138
I20260621 06:56:34.969599 6199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:35.075802 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 6138
2026-06-21T06:56:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2860 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:56:35 dist-test-slave-gmf2 krb5kdc[6669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:56:35 dist-test-slave-gmf2 krb5kdc[6669](info): set up 2 sockets
Jun 21 06:56:35 dist-test-slave-gmf2 krb5kdc[6669](info): commencing operation
krb5kdc: starting...
W20260621 06:56:36.162238 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:56:36 dist-test-slave-gmf2 krb5kdc[6669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:56:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:56:36.397118 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44853
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:44853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:36.719838 6682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:36.720115 6682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:36.720271 6682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:36.727478 6682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:56:36.727586 6682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:36.727725 6682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:36.727849 6682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:56:36.727916 6682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:56:36.737578 6682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:44853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:36.738921 6682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:36.740288 6682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:36.752444 6687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:36.752511 6690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:36.752570 6688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:36.753299 6682 server_base.cc:1034] running on GCE node
I20260621 06:56:36.754199 6682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:36.755589 6682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:36.756847 6682 hybrid_clock.cc:648] HybridClock initialized: now 1782024996756751 us; error 76 us; skew 500 ppm
I20260621 06:56:36.757227 6682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:36 dist-test-slave-gmf2 krb5kdc[6669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:36.760453 6682 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:56:36.761874 6682 webserver.cc:460] Webserver started at http://127.20.120.254:35787/ using document root <none> and password file <none>
I20260621 06:56:36.762581 6682 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:36.762679 6682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:36.762919 6682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:36.765800 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "0e8472aa494a4193894046a16648c244"
format_stamp: "Formatted at 2026-06-21 06:56:36 on dist-test-slave-gmf2"
server_key: "e3813859b77c1d148a94ca4bcf725ab3"
server_key_iv: "84ae2b7a5a1b954513bd8e92660dc67b"
server_key_version: "encryptionkey@0"
I20260621 06:56:36.766536 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "0e8472aa494a4193894046a16648c244"
format_stamp: "Formatted at 2026-06-21 06:56:36 on dist-test-slave-gmf2"
server_key: "e3813859b77c1d148a94ca4bcf725ab3"
server_key_iv: "84ae2b7a5a1b954513bd8e92660dc67b"
server_key_version: "encryptionkey@0"
I20260621 06:56:36.771328 6682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260621 06:56:36.774550 6697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:36.775741 6682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:56:36.775898 6682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "0e8472aa494a4193894046a16648c244"
format_stamp: "Formatted at 2026-06-21 06:56:36 on dist-test-slave-gmf2"
server_key: "e3813859b77c1d148a94ca4bcf725ab3"
server_key_iv: "84ae2b7a5a1b954513bd8e92660dc67b"
server_key_version: "encryptionkey@0"
I20260621 06:56:36.776081 6682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:36.795912 6682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:36.799907 6682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:36.800201 6682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:36.818483 6682 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:44853
I20260621 06:56:36.818486 6748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:44853 every 8 connection(s)
I20260621 06:56:36.819926 6682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:56:36.820268 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 6682
I20260621 06:56:36.820446 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:56:36.820958 20963 external_mini_cluster.cc:1428] Setting key c9ab12739d56373ea0bee061e5587099
I20260621 06:56:36.824360 6749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 06:56:36 dist-test-slave-gmf2 krb5kdc[6669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:56:36.837744 6749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244: Bootstrap starting.
I20260621 06:56:36.842072 6749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:36.843472 6749 log.cc:824] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:36.845587 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:36.822823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.20.120.254:44853)
0621 06:56:36.823601 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:36.823612 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:36.823654 (+ 42us) server_negotiation.cc:407] Connection header received
0621 06:56:36.825357 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:36.825378 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:36.825734 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:36.826125 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:36.827938 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:36.829285 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:36.831227 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:36.831865 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:36.831994 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:36.836913 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:36.836924 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:36.836943 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:36.836993 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:36.840939 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:36.841717 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:36.841727 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:36.841736 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:36.841986 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:36.842409 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:36.842428 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:36.842433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:36.843162 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:36.843511 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:36.843847 (+ 336us) server_negotiation.cc:299] Negotiation successful
0621 06:56:36.844351 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":360,"threads_started":1}
I20260621 06:56:36.846825 6749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244: No bootstrap required, opened a new log
I20260621 06:56:36.854748 6749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8472aa494a4193894046a16648c244" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44853 } }
I20260621 06:56:36.855100 6749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:36.855180 6749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e8472aa494a4193894046a16648c244, State: Initialized, Role: FOLLOWER
I20260621 06:56:36.855665 6749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8472aa494a4193894046a16648c244" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44853 } }
I20260621 06:56:36.855849 6749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:56:36.855953 6749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:56:36.856067 6749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:36.858157 6749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8472aa494a4193894046a16648c244" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44853 } }
I20260621 06:56:36.858570 6749 leader_election.cc:302] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e8472aa494a4193894046a16648c244; no voters:
I20260621 06:56:36.859091 6749 leader_election.cc:288] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:56:36.859256 6754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:36.860467 6754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [term 1 LEADER]: Becoming Leader. State: Replica: 0e8472aa494a4193894046a16648c244, State: Running, Role: LEADER
I20260621 06:56:36.860970 6754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8472aa494a4193894046a16648c244" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44853 } }
I20260621 06:56:36.861434 6749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:56:36.865597 6756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e8472aa494a4193894046a16648c244. Latest consensus state: current_term: 1 leader_uuid: "0e8472aa494a4193894046a16648c244" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8472aa494a4193894046a16648c244" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44853 } } }
I20260621 06:56:36.865562 6755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e8472aa494a4193894046a16648c244" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e8472aa494a4193894046a16648c244" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 44853 } } }
I20260621 06:56:36.866181 6756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:36.866178 6755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:36.867069 6761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:56:36.872433 6761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:56:36.881644 6761 catalog_manager.cc:1347] Generated new cluster ID: 3a5d3e520b0d44b3890b37d3b29d6761
I20260621 06:56:36.881819 6761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:56:36.888543 6761 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:56:36.889747 6761 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:56:36.898379 6761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244: Generated new TSK 0
I20260621 06:56:36.899189 6761 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:56:36.969715 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44853
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:37.322386 6775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:37.322669 6775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:37.322762 6775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:37.330010 6775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:37.330118 6775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:37.330236 6775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:56:37.339800 6775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:37.340999 6775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:37.342285 6775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:37.355360 6781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:37.355329 6780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:37.355594 6783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:37.356312 6775 server_base.cc:1034] running on GCE node
I20260621 06:56:37.356985 6775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:37.357888 6775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:37.359158 6775 hybrid_clock.cc:648] HybridClock initialized: now 1782024997359102 us; error 49 us; skew 500 ppm
I20260621 06:56:37.359489 6775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:37 dist-test-slave-gmf2 krb5kdc[6669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:37.363670 6775 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:56:37.365249 6775 webserver.cc:460] Webserver started at http://127.20.120.193:44187/ using document root <none> and password file <none>
I20260621 06:56:37.366086 6775 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:37.366287 6775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:37.366653 6775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:37.370443 6775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "db33ba833fc14f49acb7408646b4b90a"
format_stamp: "Formatted at 2026-06-21 06:56:37 on dist-test-slave-gmf2"
server_key: "60c7453dec71f1102e1aefcb778e592e"
server_key_iv: "541a64994625a57aa7900fa9554513c0"
server_key_version: "encryptionkey@0"
I20260621 06:56:37.371331 6775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "db33ba833fc14f49acb7408646b4b90a"
format_stamp: "Formatted at 2026-06-21 06:56:37 on dist-test-slave-gmf2"
server_key: "60c7453dec71f1102e1aefcb778e592e"
server_key_iv: "541a64994625a57aa7900fa9554513c0"
server_key_version: "encryptionkey@0"
I20260621 06:56:37.377194 6775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 06:56:37.380667 6790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:37.382014 6775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:56:37.382182 6775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "db33ba833fc14f49acb7408646b4b90a"
format_stamp: "Formatted at 2026-06-21 06:56:37 on dist-test-slave-gmf2"
server_key: "60c7453dec71f1102e1aefcb778e592e"
server_key_iv: "541a64994625a57aa7900fa9554513c0"
server_key_version: "encryptionkey@0"
I20260621 06:56:37.382375 6775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:37.401473 6775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:37.405408 6775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:37.405738 6775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:37.406766 6775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:37.408229 6775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:37.408347 6775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:37.408522 6775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:37.408617 6775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:37.448383 6775 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:41651
I20260621 06:56:37.448401 6902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:41651 every 8 connection(s)
I20260621 06:56:37.449693 6775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:56:37.458686 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 6775
I20260621 06:56:37.458886 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:56:37.459349 20963 external_mini_cluster.cc:1428] Setting key 4aed6f17c65bdb3a0430c5e15da47304
Jun 21 06:56:37 dist-test-slave-gmf2 krb5kdc[6669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:56:37.470628 6905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:37.452286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:38793 (local address 127.20.120.254:44853)
0621 06:56:37.452748 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:37.452754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:37.453708 (+ 954us) server_negotiation.cc:407] Connection header received
0621 06:56:37.455456 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:37.455460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:37.455525 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:37.455620 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:37.457025 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:37.457803 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:37.459445 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:37.460018 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:37.460158 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:37.464413 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:37.464418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:37.464421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:37.464452 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:37.467020 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:37.467991 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:37.467994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:37.467997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:37.468122 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:37.468654 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:37.468657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:37.468659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:37.468937 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:37.469090 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:37.470020 (+ 930us) server_negotiation.cc:299] Negotiation successful
0621 06:56:37.470184 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":215,"threads_started":1}
I20260621 06:56:37.471608 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:37.452874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44853 (local address 127.20.120.193:38793)
0621 06:56:37.453551 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:37.453594 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:37.455213 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:37.455815 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:37.455822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:37.456298 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:37.456878 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:37.456896 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:37.457975 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:37.457978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:37.459295 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:37.459307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:37.460238 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:37.460243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:37.460479 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:37.461205 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:56:37.461245 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:37.464191 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:37.467162 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:37.467174 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:37.467192 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:37.467838 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:37.468231 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:37.468238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:37.468244 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:37.468539 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:37.469059 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:37.469069 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:37.469543 (+ 474us) client_negotiation.cc:769] Sending connection context
0621 06:56:37.469866 (+ 323us) client_negotiation.cc:240] Negotiation successful
0621 06:56:37.470448 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":262,"threads_started":1}
I20260621 06:56:37.473242 6903 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44853
I20260621 06:56:37.473546 6903 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:37.474304 6903 heartbeater.cc:511] Master 127.20.120.254:44853 requested a full tablet report, sending...
I20260621 06:56:37.476625 6714 ts_manager.cc:194] Registered new tserver with Master: db33ba833fc14f49acb7408646b4b90a (127.20.120.193:41651)
I20260621 06:56:37.478848 6714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:38793
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:56:37.515056 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44853
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:37.830236 6909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:37.830528 6909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:37.830701 6909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:37.837636 6909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:37.837779 6909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:37.838017 6909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:56:37.847653 6909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:37.848984 6909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:37.850376 6909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:37.863338 6915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:37.863304 6914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:37.863391 6917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:37.864907 6909 server_base.cc:1034] running on GCE node
I20260621 06:56:37.865506 6909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:37.866485 6909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:37.867699 6909 hybrid_clock.cc:648] HybridClock initialized: now 1782024997867662 us; error 27 us; skew 500 ppm
I20260621 06:56:37.868002 6909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:37 dist-test-slave-gmf2 krb5kdc[6669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:37.875154 6909 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:56:37.876456 6909 webserver.cc:460] Webserver started at http://127.20.120.194:46061/ using document root <none> and password file <none>
I20260621 06:56:37.877089 6909 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:37.877208 6909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:37.877506 6909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:37.880246 6909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "54165fd5fa3244e28af338c1468add51"
format_stamp: "Formatted at 2026-06-21 06:56:37 on dist-test-slave-gmf2"
server_key: "a84136cd7513eb93f40eeb3a5aad4080"
server_key_iv: "99792c479c1b45656d4b27a4ba8c89d9"
server_key_version: "encryptionkey@0"
I20260621 06:56:37.881011 6909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "54165fd5fa3244e28af338c1468add51"
format_stamp: "Formatted at 2026-06-21 06:56:37 on dist-test-slave-gmf2"
server_key: "a84136cd7513eb93f40eeb3a5aad4080"
server_key_iv: "99792c479c1b45656d4b27a4ba8c89d9"
server_key_version: "encryptionkey@0"
I20260621 06:56:37.885951 6909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 06:56:37.889372 6924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:37.890865 6909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:56:37.891121 6909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "54165fd5fa3244e28af338c1468add51"
format_stamp: "Formatted at 2026-06-21 06:56:37 on dist-test-slave-gmf2"
server_key: "a84136cd7513eb93f40eeb3a5aad4080"
server_key_iv: "99792c479c1b45656d4b27a4ba8c89d9"
server_key_version: "encryptionkey@0"
I20260621 06:56:37.891350 6909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:37.906113 6909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:37.909777 6909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:37.910110 6909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:37.911024 6909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:37.912218 6909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:37.912319 6909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:37.912459 6909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:37.912554 6909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:37.945376 6909 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:41475
I20260621 06:56:37.945410 7036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:41475 every 8 connection(s)
I20260621 06:56:37.946699 6909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:56:37.947172 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 6909
I20260621 06:56:37.947358 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:56:37.947964 20963 external_mini_cluster.cc:1428] Setting key 826b1ce75f39c1b9de24c11070876aaa
Jun 21 06:56:37 dist-test-slave-gmf2 krb5kdc[6669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:56:37.968281 6905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:37.949463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:42571 (local address 127.20.120.254:44853)
0621 06:56:37.949632 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:37.949636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:37.951056 (+ 1420us) server_negotiation.cc:407] Connection header received
0621 06:56:37.952860 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:37.952866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:37.952945 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:37.953067 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:37.954579 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:37.955328 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:37.956848 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:37.957285 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:37.957411 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:37.961385 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:37.961394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:37.961402 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:37.961448 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:37.965119 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:37.965806 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:37.965814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:37.965821 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:37.966011 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:37.966427 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:37.966435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:37.966441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:37.966811 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:37.966956 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:37.967580 (+ 624us) server_negotiation.cc:299] Negotiation successful
0621 06:56:37.967827 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 06:56:37.969024 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:37.950169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44853 (local address 127.20.120.194:42571)
0621 06:56:37.950908 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:37.950952 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:37.952594 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:37.953294 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:37.953302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:37.953787 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:37.954389 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:37.954406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:37.955496 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:37.955500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:37.956695 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:37.956706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:37.957517 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:37.957524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:37.957743 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:37.958501 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:56:37.958533 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:37.961139 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:37.965257 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:37.965263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:37.965274 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:37.965674 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:37.966118 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:37.966122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:37.966124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:37.966305 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:37.966936 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:37.966941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:37.967219 (+ 278us) client_negotiation.cc:769] Sending connection context
0621 06:56:37.967421 (+ 202us) client_negotiation.cc:240] Negotiation successful
0621 06:56:37.967825 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":319,"threads_started":1}
I20260621 06:56:37.970453 7037 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44853
I20260621 06:56:37.970796 7037 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:37.971704 7037 heartbeater.cc:511] Master 127.20.120.254:44853 requested a full tablet report, sending...
I20260621 06:56:37.973284 6714 ts_manager.cc:194] Registered new tserver with Master: 54165fd5fa3244e28af338c1468add51 (127.20.120.194:41475)
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
I20260621 06:56:37.974161 6714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:42571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:56:38.003036 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44853
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:38.315464 7042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:38.315757 7042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:38.315919 7042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:38.323240 7042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:38.323383 7042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:38.323575 7042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:56:38.332708 7042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:44853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:38.333926 7042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:38.335214 7042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:38.347688 7050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:38.347769 7048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:38.347796 7047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:38.348675 7042 server_base.cc:1034] running on GCE node
I20260621 06:56:38.349324 7042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:38.350234 7042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:38.351428 7042 hybrid_clock.cc:648] HybridClock initialized: now 1782024998351395 us; error 31 us; skew 500 ppm
I20260621 06:56:38.351725 7042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:38 dist-test-slave-gmf2 krb5kdc[6669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024998, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:38.354651 7042 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:56:38.356017 7042 webserver.cc:460] Webserver started at http://127.20.120.195:38189/ using document root <none> and password file <none>
I20260621 06:56:38.356631 7042 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:38.356752 7042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:38.357067 7042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:38.359818 7042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "2612dbe31f0c4e9b813af03f287b71f4"
format_stamp: "Formatted at 2026-06-21 06:56:38 on dist-test-slave-gmf2"
server_key: "a9e79da5dd413f4cb93e44451e203b0d"
server_key_iv: "0942a5c0abad1d7842b7d85c00037a80"
server_key_version: "encryptionkey@0"
I20260621 06:56:38.360520 7042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "2612dbe31f0c4e9b813af03f287b71f4"
format_stamp: "Formatted at 2026-06-21 06:56:38 on dist-test-slave-gmf2"
server_key: "a9e79da5dd413f4cb93e44451e203b0d"
server_key_iv: "0942a5c0abad1d7842b7d85c00037a80"
server_key_version: "encryptionkey@0"
I20260621 06:56:38.365490 7042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260621 06:56:38.368860 7057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:38.370198 7042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:56:38.370349 7042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "2612dbe31f0c4e9b813af03f287b71f4"
format_stamp: "Formatted at 2026-06-21 06:56:38 on dist-test-slave-gmf2"
server_key: "a9e79da5dd413f4cb93e44451e203b0d"
server_key_iv: "0942a5c0abad1d7842b7d85c00037a80"
server_key_version: "encryptionkey@0"
I20260621 06:56:38.370528 7042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:38.401839 7042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:38.405716 7042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:38.406038 7042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:38.406980 7042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:38.408250 7042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:38.408321 7042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:38.408492 7042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:38.408545 7042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:38.440582 7042 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:44383
I20260621 06:56:38.440671 7169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:44383 every 8 connection(s)
I20260621 06:56:38.441891 7042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:56:38.443805 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 7042
I20260621 06:56:38.444007 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:56:38.444568 20963 external_mini_cluster.cc:1428] Setting key 83cdb78ff76b156693146e6f340a1127
Jun 21 06:56:38 dist-test-slave-gmf2 krb5kdc[6669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024998, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:56:38.462642 6905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.444543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:39187 (local address 127.20.120.254:44853)
0621 06:56:38.444762 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:38.444768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:38.445938 (+ 1170us) server_negotiation.cc:407] Connection header received
0621 06:56:38.448224 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:38.448230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:38.448312 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:38.448431 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:38.449847 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.450638 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.452205 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.452607 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.452741 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.456125 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:38.456131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:38.456135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:38.456169 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:38.459429 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:38.460333 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:38.460337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:38.460339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:38.460504 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:38.460886 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:38.460890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:38.460896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:38.461187 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:38.461334 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:38.462126 (+ 792us) server_negotiation.cc:299] Negotiation successful
0621 06:56:38.462337 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 06:56:38.463675 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.444990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:44853 (local address 127.20.120.195:39187)
0621 06:56:38.445744 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:38.445815 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:38.447947 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:38.448660 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:38.448668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:38.449141 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:38.449687 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:38.449703 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.450789 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.450799 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:38.452064 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:38.452075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.452827 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.452836 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:38.453058 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.453663 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:56:38.453693 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:38.455924 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:38.459600 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:38.459606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:38.459617 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:38.460212 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:38.460606 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:38.460609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:38.460612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:38.460780 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:38.461383 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:38.461392 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:38.461719 (+ 327us) client_negotiation.cc:769] Sending connection context
0621 06:56:38.462034 (+ 315us) client_negotiation.cc:240] Negotiation successful
0621 06:56:38.462511 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":305,"threads_started":1}
I20260621 06:56:38.464952 7170 heartbeater.cc:347] Connected to a master server at 127.20.120.254:44853
I20260621 06:56:38.465212 7170 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:38.465874 7170 heartbeater.cc:511] Master 127.20.120.254:44853 requested a full tablet report, sending...
I20260621 06:56:38.467278 6714 ts_manager.cc:194] Registered new tserver with Master: 2612dbe31f0c4e9b813af03f287b71f4 (127.20.120.195:44383)
I20260621 06:56:38.468123 6714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:39187
I20260621 06:56:38.472237 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:56:38.482088 6903 heartbeater.cc:503] Master 127.20.120.254:44853 was elected leader, sending a full tablet report...
I20260621 06:56:38.489811 6905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.476591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.20.120.254:44853)
0621 06:56:38.476747 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:38.476752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:38.477337 (+ 585us) server_negotiation.cc:407] Connection header received
0621 06:56:38.477493 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:38.477496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:38.477550 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:38.477628 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:38.479323 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.480014 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.481461 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.481966 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.482061 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.484909 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:38.484913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:38.484917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:38.484945 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:38.487460 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:38.488096 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:38.488099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:38.488101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:38.488213 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:38.488576 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:38.488579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:38.488581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:38.488855 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:38.489053 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:38.489363 (+ 310us) server_negotiation.cc:299] Negotiation successful
0621 06:56:38.489483 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 06:56:38.493431 6714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:56:38.500250 6714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:56:38.527822 7182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.511921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:41475 (local address 127.0.0.1:35596)
0621 06:56:38.512897 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:38.512946 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:38.513124 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:38.514215 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:38.514226 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:38.514268 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:38.514800 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:38.514815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.518801 (+ 3986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.520100 (+ 1299us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:38.525094 (+ 4994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:38.525109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.526765 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.526771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:38.527009 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.527066 (+ 57us) client_negotiation.cc:769] Sending connection context
0621 06:56:38.527216 (+ 150us) client_negotiation.cc:240] Negotiation successful
0621 06:56:38.527460 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":435,"threads_started":1}
I20260621 06:56:38.528378 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.512110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.20.120.194:41475)
0621 06:56:38.513372 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:38.513379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:38.513401 (+ 22us) server_negotiation.cc:407] Connection header received
0621 06:56:38.513478 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:38.513484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:38.513715 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:38.513943 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:38.516439 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.518574 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.525280 (+ 6706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.526533 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.526699 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.526875 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:38.527661 (+ 786us) server_negotiation.cc:299] Negotiation successful
0621 06:56:38.527836 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":962,"threads_started":1}
I20260621 06:56:38.532617 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.513257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:44383 (local address 127.0.0.1:39084)
0621 06:56:38.513731 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:38.513748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:38.513863 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:38.526411 (+ 12548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:38.526415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:38.526435 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:38.526779 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:38.526789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.528622 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.528627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:38.530677 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:38.530689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.531894 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.531900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:38.532056 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.532090 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 06:56:38.532176 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 06:56:38.532308 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":374,"threads_started":1}
I20260621 06:56:38.532735 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.513899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.20.120.195:44383)
0621 06:56:38.525379 (+ 11480us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:38.525391 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:38.525432 (+ 41us) server_negotiation.cc:407] Connection header received
0621 06:56:38.525549 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:38.525560 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:38.525880 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:38.526169 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:38.527005 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.528478 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.530841 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.531713 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.531908 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.532066 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:38.532256 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 06:56:38.532404 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11233,"thread_start_us":11049,"threads_started":1}
I20260621 06:56:38.532255 6972 tablet_service.cc:1467] Processing CreateTablet for tablet 1c45dfab3ed34916b48a915c6f0706d6 (DEFAULT_TABLE table=test-table [id=f96738785b5f497493562e12caeaf892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:38.533893 6972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c45dfab3ed34916b48a915c6f0706d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:38.534164 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.514157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:41651 (local address 127.0.0.1:52512)
0621 06:56:38.524339 (+ 10182us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:38.524363 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:38.524505 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:38.527041 (+ 2536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:38.527045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:38.527069 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:38.527442 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:38.527453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.529541 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.529546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:38.531596 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:38.531611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.533402 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.533409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:38.533563 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.533602 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 06:56:38.533714 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 06:56:38.533829 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9991,"thread_start_us":10447,"threads_started":1}
I20260621 06:56:38.534246 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.514348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52512 (local address 127.20.120.193:41651)
0621 06:56:38.526444 (+ 12096us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:38.526452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:38.526480 (+ 28us) server_negotiation.cc:407] Connection header received
0621 06:56:38.526576 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:38.526586 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:38.526806 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:38.527046 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:38.527603 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.529332 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.531768 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.533195 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.533417 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.533567 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:38.533755 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 06:56:38.533932 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11912,"thread_start_us":11729,"threads_started":1}
I20260621 06:56:38.536329 7105 tablet_service.cc:1467] Processing CreateTablet for tablet 1c45dfab3ed34916b48a915c6f0706d6 (DEFAULT_TABLE table=test-table [id=f96738785b5f497493562e12caeaf892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:38.537722 7105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c45dfab3ed34916b48a915c6f0706d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:38.537358 6838 tablet_service.cc:1467] Processing CreateTablet for tablet 1c45dfab3ed34916b48a915c6f0706d6 (DEFAULT_TABLE table=test-table [id=f96738785b5f497493562e12caeaf892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:38.538482 6838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c45dfab3ed34916b48a915c6f0706d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:38.548418 7188 tablet_bootstrap.cc:492] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51: Bootstrap starting.
I20260621 06:56:38.550108 7189 tablet_bootstrap.cc:492] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a: Bootstrap starting.
I20260621 06:56:38.552732 7188 tablet_bootstrap.cc:654] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:38.553683 7188 log.cc:824] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:38.554162 7189 tablet_bootstrap.cc:654] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:38.554550 7190 tablet_bootstrap.cc:492] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4: Bootstrap starting.
I20260621 06:56:38.555397 7189 log.cc:824] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:38.558043 7190 tablet_bootstrap.cc:654] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:38.558175 7188 tablet_bootstrap.cc:492] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51: No bootstrap required, opened a new log
I20260621 06:56:38.558423 7188 ts_tablet_manager.cc:1397] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260621 06:56:38.558436 7189 tablet_bootstrap.cc:492] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a: No bootstrap required, opened a new log
I20260621 06:56:38.558683 7189 ts_tablet_manager.cc:1397] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 06:56:38.559177 7190 log.cc:824] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:38.562044 7190 tablet_bootstrap.cc:492] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4: No bootstrap required, opened a new log
I20260621 06:56:38.562335 7190 ts_tablet_manager.cc:1397] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260621 06:56:38.566246 7188 raft_consensus.cc:348] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54165fd5fa3244e28af338c1468add51" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 41475 } } peers { permanent_uuid: "2612dbe31f0c4e9b813af03f287b71f4" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 44383 } } peers { permanent_uuid: "db33ba833fc14f49acb7408646b4b90a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 41651 } }
I20260621 06:56:38.566597 7188 raft_consensus.cc:374] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:38.566721 7188 raft_consensus.cc:729] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54165fd5fa3244e28af338c1468add51, State: Initialized, Role: FOLLOWER
I20260621 06:56:38.567276 7188 consensus_queue.cc:260] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54165fd5fa3244e28af338c1468add51" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 41475 } } peers { permanent_uuid: "2612dbe31f0c4e9b813af03f287b71f4" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 44383 } } peers { permanent_uuid: "db33ba833fc14f49acb7408646b4b90a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 41651 } }
I20260621 06:56:38.568367 7037 heartbeater.cc:503] Master 127.20.120.254:44853 was elected leader, sending a full tablet report...
I20260621 06:56:38.569079 7188 ts_tablet_manager.cc:1428] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260621 06:56:38.568806 7189 raft_consensus.cc:348] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54165fd5fa3244e28af338c1468add51" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 41475 } } peers { permanent_uuid: "2612dbe31f0c4e9b813af03f287b71f4" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 44383 } } peers { permanent_uuid: "db33ba833fc14f49acb7408646b4b90a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 41651 } }
I20260621 06:56:38.569195 7189 raft_consensus.cc:374] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:38.569344 7189 raft_consensus.cc:729] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db33ba833fc14f49acb7408646b4b90a, State: Initialized, Role: FOLLOWER
I20260621 06:56:38.569988 7189 consensus_queue.cc:260] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54165fd5fa3244e28af338c1468add51" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 41475 } } peers { permanent_uuid: "2612dbe31f0c4e9b813af03f287b71f4" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 44383 } } peers { permanent_uuid: "db33ba833fc14f49acb7408646b4b90a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 41651 } }
I20260621 06:56:38.571552 7189 ts_tablet_manager.cc:1428] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260621 06:56:38.572535 7190 raft_consensus.cc:348] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54165fd5fa3244e28af338c1468add51" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 41475 } } peers { permanent_uuid: "2612dbe31f0c4e9b813af03f287b71f4" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 44383 } } peers { permanent_uuid: "db33ba833fc14f49acb7408646b4b90a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 41651 } }
I20260621 06:56:38.572928 7190 raft_consensus.cc:374] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:38.573067 7190 raft_consensus.cc:729] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2612dbe31f0c4e9b813af03f287b71f4, State: Initialized, Role: FOLLOWER
I20260621 06:56:38.573819 7190 consensus_queue.cc:260] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54165fd5fa3244e28af338c1468add51" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 41475 } } peers { permanent_uuid: "2612dbe31f0c4e9b813af03f287b71f4" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 44383 } } peers { permanent_uuid: "db33ba833fc14f49acb7408646b4b90a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 41651 } }
I20260621 06:56:38.575443 7170 heartbeater.cc:503] Master 127.20.120.254:44853 was elected leader, sending a full tablet report...
I20260621 06:56:38.575893 7190 ts_tablet_manager.cc:1428] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260621 06:56:38.693508 7171 tablet.cc:2369] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:56:38.698642 7038 tablet.cc:2369] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:56:38.702020 6904 tablet.cc:2369] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:56:38.773080 7194 raft_consensus.cc:482] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:56:38.773448 7194 raft_consensus.cc:504] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54165fd5fa3244e28af338c1468add51" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 41475 } } peers { permanent_uuid: "2612dbe31f0c4e9b813af03f287b71f4" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 44383 } } peers { permanent_uuid: "db33ba833fc14f49acb7408646b4b90a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 41651 } }
I20260621 06:56:38.775595 7194 leader_election.cc:288] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2612dbe31f0c4e9b813af03f287b71f4 (127.20.120.195:44383), db33ba833fc14f49acb7408646b4b90a (127.20.120.193:41651)
I20260621 06:56:38.783272 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.775923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:34181 (local address 127.20.120.195:44383)
0621 06:56:38.776109 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:38.776113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:38.776717 (+ 604us) server_negotiation.cc:407] Connection header received
0621 06:56:38.776991 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:38.777001 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:38.777085 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:38.777151 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:38.777773 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.778794 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.781327 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.782489 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.782648 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.782725 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:38.782939 (+ 214us) server_negotiation.cc:299] Negotiation successful
0621 06:56:38.783072 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 06:56:38.783399 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.775913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:44383 (local address 127.20.120.194:34181)
0621 06:56:38.776605 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:38.776622 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:38.776758 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:38.777333 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:38.777337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:38.777357 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:38.777642 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:38.777650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.778920 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.778923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:38.781181 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:38.781195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.782673 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.782677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:38.782792 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.782814 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 06:56:38.782888 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 06:56:38.783024 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":361,"threads_started":1}
I20260621 06:56:38.783823 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.776892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:51157 (local address 127.20.120.193:41651)
0621 06:56:38.777122 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:38.777127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:38.777393 (+ 266us) server_negotiation.cc:407] Connection header received
0621 06:56:38.777602 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:38.777613 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:38.777669 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:38.777772 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:38.778269 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.779243 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.781679 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.782907 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.783049 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.783109 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:38.783535 (+ 426us) server_negotiation.cc:299] Negotiation successful
0621 06:56:38.783666 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260621 06:56:38.784322 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.776684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:41651 (local address 127.20.120.194:51157)
0621 06:56:38.777283 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:38.777296 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:38.777469 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:38.777806 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:38.777809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:38.777826 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:38.778139 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:38.778146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.779376 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.779382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:38.781501 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:38.781515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.783118 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.783126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:38.783310 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.783343 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 06:56:38.784003 (+ 660us) client_negotiation.cc:240] Negotiation successful
0621 06:56:38.784088 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":1,"thread_start_us":313,"threads_started":1}
I20260621 06:56:38.784358 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c45dfab3ed34916b48a915c6f0706d6" candidate_uuid: "54165fd5fa3244e28af338c1468add51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2612dbe31f0c4e9b813af03f287b71f4" is_pre_election: true
I20260621 06:56:38.784863 7125 raft_consensus.cc:2454] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54165fd5fa3244e28af338c1468add51 in term 0.
I20260621 06:56:38.785377 6858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c45dfab3ed34916b48a915c6f0706d6" candidate_uuid: "54165fd5fa3244e28af338c1468add51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db33ba833fc14f49acb7408646b4b90a" is_pre_election: true
I20260621 06:56:38.785586 6928 leader_election.cc:302] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2612dbe31f0c4e9b813af03f287b71f4, 54165fd5fa3244e28af338c1468add51; no voters:
I20260621 06:56:38.785974 7194 raft_consensus.cc:2781] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:56:38.785975 6858 raft_consensus.cc:2454] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54165fd5fa3244e28af338c1468add51 in term 0.
I20260621 06:56:38.786123 7194 raft_consensus.cc:482] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:56:38.786240 7194 raft_consensus.cc:3037] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:38.788352 7194 raft_consensus.cc:504] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54165fd5fa3244e28af338c1468add51" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 41475 } } peers { permanent_uuid: "2612dbe31f0c4e9b813af03f287b71f4" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 44383 } } peers { permanent_uuid: "db33ba833fc14f49acb7408646b4b90a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 41651 } }
I20260621 06:56:38.788909 7194 leader_election.cc:288] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [CANDIDATE]: Term 1 election: Requested vote from peers 2612dbe31f0c4e9b813af03f287b71f4 (127.20.120.195:44383), db33ba833fc14f49acb7408646b4b90a (127.20.120.193:41651)
I20260621 06:56:38.789297 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c45dfab3ed34916b48a915c6f0706d6" candidate_uuid: "54165fd5fa3244e28af338c1468add51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2612dbe31f0c4e9b813af03f287b71f4"
I20260621 06:56:38.789367 6858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c45dfab3ed34916b48a915c6f0706d6" candidate_uuid: "54165fd5fa3244e28af338c1468add51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db33ba833fc14f49acb7408646b4b90a"
I20260621 06:56:38.789564 6858 raft_consensus.cc:3037] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:38.789572 7125 raft_consensus.cc:3037] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:38.791822 6858 raft_consensus.cc:2454] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54165fd5fa3244e28af338c1468add51 in term 1.
I20260621 06:56:38.792088 7125 raft_consensus.cc:2454] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54165fd5fa3244e28af338c1468add51 in term 1.
I20260621 06:56:38.792281 6928 leader_election.cc:302] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54165fd5fa3244e28af338c1468add51, db33ba833fc14f49acb7408646b4b90a; no voters:
I20260621 06:56:38.792609 7194 raft_consensus.cc:2781] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:38.793097 7194 raft_consensus.cc:686] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 1 LEADER]: Becoming Leader. State: Replica: 54165fd5fa3244e28af338c1468add51, State: Running, Role: LEADER
I20260621 06:56:38.793603 7194 consensus_queue.cc:237] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54165fd5fa3244e28af338c1468add51" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 41475 } } peers { permanent_uuid: "2612dbe31f0c4e9b813af03f287b71f4" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 44383 } } peers { permanent_uuid: "db33ba833fc14f49acb7408646b4b90a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 41651 } }
I20260621 06:56:38.798403 6713 catalog_manager.cc:5495] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54165fd5fa3244e28af338c1468add51 (127.20.120.194). New cstate: current_term: 1 leader_uuid: "54165fd5fa3244e28af338c1468add51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54165fd5fa3244e28af338c1468add51" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 41475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2612dbe31f0c4e9b813af03f287b71f4" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 44383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db33ba833fc14f49acb7408646b4b90a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20260621 06:56:38.816278 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.811015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.20.120.194:41475)
0621 06:56:38.811187 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:38.811191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:38.811208 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:56:38.811322 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:38.811325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:38.811401 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:38.811519 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:56:38.811929 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.812663 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.814242 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.814633 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.814734 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.815181 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:56:38.815667 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:56:38.815793 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:38.815932 (+ 139us) server_negotiation.cc:299] Negotiation successful
0621 06:56:38.815997 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 06:56:38.824818 7125 raft_consensus.cc:1261] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4 [term 1 FOLLOWER]: Refusing update from remote peer 54165fd5fa3244e28af338c1468add51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:56:38.824818 6858 raft_consensus.cc:1261] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a [term 1 FOLLOWER]: Refusing update from remote peer 54165fd5fa3244e28af338c1468add51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:56:38.825819 7194 consensus_queue.cc:1035] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db33ba833fc14f49acb7408646b4b90a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:56:38.826165 7199 consensus_queue.cc:1035] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2612dbe31f0c4e9b813af03f287b71f4" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:56:38.852694 6714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:56:38.853003 6714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:56:38.856487 6714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e8472aa494a4193894046a16648c244: Sending DeleteTablet for 3 replicas of tablet 1c45dfab3ed34916b48a915c6f0706d6
I20260621 06:56:38.857575 6972 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c45dfab3ed34916b48a915c6f0706d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:38 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:35596
I20260621 06:56:38.857801 7105 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c45dfab3ed34916b48a915c6f0706d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:38 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:39084
I20260621 06:56:38.858033 6838 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c45dfab3ed34916b48a915c6f0706d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:38 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:52512
I20260621 06:56:38.862366 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 6775
I20260621 06:56:38.870263 7211 tablet_replica.cc:330] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51: stopping tablet replica
I20260621 06:56:38.870956 7213 tablet_replica.cc:330] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a: stopping tablet replica
I20260621 06:56:38.871549 7212 tablet_replica.cc:330] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4: stopping tablet replica
I20260621 06:56:38.871836 7211 raft_consensus.cc:2229] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:56:38.872359 7213 raft_consensus.cc:2229] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:56:38.872594 7211 raft_consensus.cc:2258] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:38.872543 7212 raft_consensus.cc:2229] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:56:38.873517 7213 raft_consensus.cc:2258] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:38.873869 7212 raft_consensus.cc:2258] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:38.875895 7211 ts_tablet_manager.cc:1905] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:38.877779 7213 ts_tablet_manager.cc:1905] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:38.878814 7212 ts_tablet_manager.cc:1905] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 06:56:38 dist-test-slave-gmf2 krb5kdc[6669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:56:38.884739 7213 ts_tablet_manager.cc:1918] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:38.884938 7213 log.cc:1196] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/1c45dfab3ed34916b48a915c6f0706d6
I20260621 06:56:38.885473 7213 ts_tablet_manager.cc:1939] T 1c45dfab3ed34916b48a915c6f0706d6 P db33ba833fc14f49acb7408646b4b90a: Deleting consensus metadata
I20260621 06:56:38.887907 6701 catalog_manager.cc:4841] TS db33ba833fc14f49acb7408646b4b90a (127.20.120.193:41651): tablet 1c45dfab3ed34916b48a915c6f0706d6 (table test-table [id=f96738785b5f497493562e12caeaf892]) successfully deleted
I20260621 06:56:38.889503 7211 ts_tablet_manager.cc:1918] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:38.889717 7211 log.cc:1196] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/1c45dfab3ed34916b48a915c6f0706d6
I20260621 06:56:38.890221 7212 ts_tablet_manager.cc:1918] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:38.890326 7211 ts_tablet_manager.cc:1939] T 1c45dfab3ed34916b48a915c6f0706d6 P 54165fd5fa3244e28af338c1468add51: Deleting consensus metadata
I20260621 06:56:38.890424 7212 log.cc:1196] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/1c45dfab3ed34916b48a915c6f0706d6
I20260621 06:56:38.890911 7212 ts_tablet_manager.cc:1939] T 1c45dfab3ed34916b48a915c6f0706d6 P 2612dbe31f0c4e9b813af03f287b71f4: Deleting consensus metadata
I20260621 06:56:38.892074 6701 catalog_manager.cc:4841] TS 54165fd5fa3244e28af338c1468add51 (127.20.120.194:41475): tablet 1c45dfab3ed34916b48a915c6f0706d6 (table test-table [id=f96738785b5f497493562e12caeaf892]) successfully deleted
I20260621 06:56:38.892489 6701 catalog_manager.cc:4841] TS 2612dbe31f0c4e9b813af03f287b71f4 (127.20.120.195:44383): tablet 1c45dfab3ed34916b48a915c6f0706d6 (table test-table [id=f96738785b5f497493562e12caeaf892]) successfully deleted
I20260621 06:56:38.893339 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:38.863665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.20.120.193:41651)
0621 06:56:38.863848 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:38.863854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:38.873385 (+ 9531us) server_negotiation.cc:407] Connection header received
0621 06:56:38.873743 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:38.873749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:38.873836 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:38.873993 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:38.875558 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.876782 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.878770 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:38.879441 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:38.879548 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:38.885372 (+ 5824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:38.885388 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:38.885406 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:38.885447 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:38.889988 (+ 4541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:38.890874 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:38.890883 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:38.890886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:38.891042 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:38.891520 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:38.891526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:38.891530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:38.891971 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:38.892244 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:38.892927 (+ 683us) server_negotiation.cc:299] Negotiation successful
0621 06:56:38.893099 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:56:38.894009 6898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:39.043349 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 6775
I20260621 06:56:39.061326 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 6909
Jun 21 06:56:39 dist-test-slave-gmf2 krb5kdc[6669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:56:39.079356 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:39.062106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.20.120.194:41475)
0621 06:56:39.062305 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:39.062310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:39.062409 (+ 99us) server_negotiation.cc:407] Connection header received
0621 06:56:39.062625 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:39.062630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:39.062714 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:39.062837 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:39.064485 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:39.065700 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:39.067093 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:39.067790 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:39.067916 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:39.072580 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:39.072598 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:39.072620 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:39.072666 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:39.076544 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:39.077224 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:39.077233 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:39.077237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:39.077412 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:39.077755 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:39.077760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:39.077763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:39.078218 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:39.078547 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:39.078721 (+ 174us) server_negotiation.cc:299] Negotiation successful
0621 06:56:39.078942 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:56:39.080168 7032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:39.193089 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 6909
I20260621 06:56:39.212121 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 7042
Jun 21 06:56:39 dist-test-slave-gmf2 krb5kdc[6669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782024996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:56:39.226893 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:39.212952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.20.120.195:44383)
0621 06:56:39.213165 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:39.213171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:39.213192 (+ 21us) server_negotiation.cc:407] Connection header received
0621 06:56:39.213428 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:39.213433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:39.213495 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:39.213604 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:39.214943 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:39.215644 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:39.216930 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:39.217326 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:39.217457 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:39.221942 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:39.221965 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:39.221989 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:39.222050 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:39.224568 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:39.225157 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:39.225163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:39.225166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:39.225301 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:39.225603 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:39.225606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:39.225608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:39.225997 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:39.226213 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:39.226478 (+ 265us) server_negotiation.cc:299] Negotiation successful
0621 06:56:39.226629 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 06:56:39.227535 7165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:39.316175 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 7042
I20260621 06:56:39.332721 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 6682
I20260621 06:56:39.333783 6744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:39.450054 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 6682
2026-06-21T06:56:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4378 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-21T06:56:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:56:39Z Disabled control of system clock
I20260621 06:56:39.525764 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34029
--webserver_interface=127.20.120.254
--webserver_port=0
--builtin_ntp_servers=127.20.120.212:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:34029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 06:56:39.850003 7222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:39.850303 7222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:39.850386 7222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:39.857686 7222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:56:39.857800 7222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:39.857837 7222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:39.857893 7222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:56:39.857959 7222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:56:39.867419 7222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:34029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:34029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:39.868701 7222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:39.870445 7222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:39.882283 7227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:39.882221 7228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:39.882184 7230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:39.883169 7222 server_base.cc:1034] running on GCE node
I20260621 06:56:39.883823 7222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:39.885205 7222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:39.886394 7222 hybrid_clock.cc:648] HybridClock initialized: now 1782024999886337 us; error 53 us; skew 500 ppm
I20260621 06:56:39.886734 7222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:56:39.888895 7222 webserver.cc:460] Webserver started at http://127.20.120.254:46671/ using document root <none> and password file <none>
I20260621 06:56:39.889554 7222 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:39.889664 7222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:39.890070 7222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:39.892846 7222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "515995b1048a40c28f568b1e3d17e8fa"
format_stamp: "Formatted at 2026-06-21 06:56:39 on dist-test-slave-gmf2"
server_key: "8ed685b50c29e011208f646179d6a52e"
server_key_iv: "ba2318a1ce1ac55963fd48c0f7080ad8"
server_key_version: "encryptionkey@0"
I20260621 06:56:39.893668 7222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "515995b1048a40c28f568b1e3d17e8fa"
format_stamp: "Formatted at 2026-06-21 06:56:39 on dist-test-slave-gmf2"
server_key: "8ed685b50c29e011208f646179d6a52e"
server_key_iv: "ba2318a1ce1ac55963fd48c0f7080ad8"
server_key_version: "encryptionkey@0"
I20260621 06:56:39.898674 7222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260621 06:56:39.901851 7236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:39.903208 7222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:56:39.903393 7222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "515995b1048a40c28f568b1e3d17e8fa"
format_stamp: "Formatted at 2026-06-21 06:56:39 on dist-test-slave-gmf2"
server_key: "8ed685b50c29e011208f646179d6a52e"
server_key_iv: "ba2318a1ce1ac55963fd48c0f7080ad8"
server_key_version: "encryptionkey@0"
I20260621 06:56:39.903604 7222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:39.921799 7222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:39.922608 7222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:39.922891 7222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:39.941303 7222 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:34029
I20260621 06:56:39.941378 7287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:34029 every 8 connection(s)
I20260621 06:56:39.943037 7222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:56:39.945983 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 7222
I20260621 06:56:39.946161 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:56:39.946707 20963 external_mini_cluster.cc:1428] Setting key a4fcaf9f2603ca3b0aa54e4b53fc8f04
I20260621 06:56:39.947190 7288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:39.961939 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:39.948821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.20.120.254:34029)
0621 06:56:39.950072 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:39.950092 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:39.950143 (+ 51us) server_negotiation.cc:407] Connection header received
0621 06:56:39.952038 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:39.952066 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:39.952660 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:39.953092 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:39.953621 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:39.955615 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:39.956972 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:39.958009 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:39.958192 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:39.959275 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:39.959286 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:39.959302 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:39.960014 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:39.960108 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:39.960137 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:39.960277 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:39.960394 (+ 117us) server_negotiation.cc:299] Negotiation successful
0621 06:56:39.960695 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":422,"threads_started":1}
I20260621 06:56:39.962406 7288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa: Bootstrap starting.
I20260621 06:56:39.965303 7288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:39.966406 7288 log.cc:824] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:39.968887 7288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa: No bootstrap required, opened a new log
I20260621 06:56:39.975760 7288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515995b1048a40c28f568b1e3d17e8fa" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34029 } }
I20260621 06:56:39.976102 7288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:39.976207 7288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515995b1048a40c28f568b1e3d17e8fa, State: Initialized, Role: FOLLOWER
I20260621 06:56:39.976727 7288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515995b1048a40c28f568b1e3d17e8fa" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34029 } }
I20260621 06:56:39.976966 7288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:56:39.977109 7288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:56:39.977288 7288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:39.979316 7288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515995b1048a40c28f568b1e3d17e8fa" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34029 } }
I20260621 06:56:39.979780 7288 leader_election.cc:302] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 515995b1048a40c28f568b1e3d17e8fa; no voters:
I20260621 06:56:39.980319 7288 leader_election.cc:288] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:56:39.980460 7293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:39.981161 7293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [term 1 LEADER]: Becoming Leader. State: Replica: 515995b1048a40c28f568b1e3d17e8fa, State: Running, Role: LEADER
I20260621 06:56:39.981674 7293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515995b1048a40c28f568b1e3d17e8fa" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34029 } }
I20260621 06:56:39.982398 7288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:56:39.984627 7294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "515995b1048a40c28f568b1e3d17e8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515995b1048a40c28f568b1e3d17e8fa" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34029 } } }
I20260621 06:56:39.984625 7295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 515995b1048a40c28f568b1e3d17e8fa. Latest consensus state: current_term: 1 leader_uuid: "515995b1048a40c28f568b1e3d17e8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515995b1048a40c28f568b1e3d17e8fa" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 34029 } } }
I20260621 06:56:39.985072 7294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:39.985142 7295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:39.987128 7300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:56:39.992156 7300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:56:40.002509 7300 catalog_manager.cc:1347] Generated new cluster ID: 4403ade6f894403b8e171d7d97e2e0d6
I20260621 06:56:40.002635 7300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:56:40.018913 7300 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:56:40.020248 7300 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:56:40.026401 7300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa: Generated new TSK 0
I20260621 06:56:40.027311 7300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:56:40.040548 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34029
--builtin_ntp_servers=127.20.120.212:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 06:56:40.366117 7312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:40.366379 7312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:40.366540 7312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:40.373582 7312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:40.373670 7312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:40.373896 7312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:56:40.382772 7312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:40.384177 7312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:40.385975 7312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:40.399008 7318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:40.399315 7317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:40.399714 7320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:40.399775 7312 server_base.cc:1034] running on GCE node
I20260621 06:56:40.400393 7312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:40.401280 7312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:40.402513 7312 hybrid_clock.cc:648] HybridClock initialized: now 1782025000402463 us; error 61 us; skew 500 ppm
I20260621 06:56:40.402824 7312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:56:40.404999 7312 webserver.cc:460] Webserver started at http://127.20.120.193:44037/ using document root <none> and password file <none>
I20260621 06:56:40.405659 7312 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:40.405745 7312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:40.406105 7312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:40.409036 7312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "111ec1795b664573a60df62f233e39f6"
format_stamp: "Formatted at 2026-06-21 06:56:40 on dist-test-slave-gmf2"
server_key: "54999be96aa544296e2d93854a130f8a"
server_key_iv: "e3d79a2007ccd54da16b366c77f168ed"
server_key_version: "encryptionkey@0"
I20260621 06:56:40.409859 7312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "111ec1795b664573a60df62f233e39f6"
format_stamp: "Formatted at 2026-06-21 06:56:40 on dist-test-slave-gmf2"
server_key: "54999be96aa544296e2d93854a130f8a"
server_key_iv: "e3d79a2007ccd54da16b366c77f168ed"
server_key_version: "encryptionkey@0"
I20260621 06:56:40.414886 7312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260621 06:56:40.418280 7326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:40.419584 7312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 06:56:40.419757 7312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "111ec1795b664573a60df62f233e39f6"
format_stamp: "Formatted at 2026-06-21 06:56:40 on dist-test-slave-gmf2"
server_key: "54999be96aa544296e2d93854a130f8a"
server_key_iv: "e3d79a2007ccd54da16b366c77f168ed"
server_key_version: "encryptionkey@0"
I20260621 06:56:40.419965 7312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:40.445205 7312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:40.446115 7312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:40.446430 7312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:40.447399 7312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:40.448832 7312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:40.448941 7312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:40.449070 7312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:40.449182 7312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:40.481801 7312 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:35805
I20260621 06:56:40.481890 7438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:35805 every 8 connection(s)
I20260621 06:56:40.483253 7312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:56:40.493546 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 7312
I20260621 06:56:40.493718 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:56:40.494333 20963 external_mini_cluster.cc:1428] Setting key 7eb3b1c3408f6e034407b9af603925a0
I20260621 06:56:40.497792 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34029
--builtin_ntp_servers=127.20.120.212:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 06:56:40.498241 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:40.485838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:56845 (local address 127.20.120.254:34029)
0621 06:56:40.486389 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:40.486396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:40.487456 (+ 1060us) server_negotiation.cc:407] Connection header received
0621 06:56:40.489959 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:40.489966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:40.490047 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:40.490164 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:40.491278 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:40.492626 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:40.495031 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:40.495806 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:40.495964 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:40.497437 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:40.497442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:40.497445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:40.497461 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:40.497496 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:40.497508 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:40.497578 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:40.497808 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 06:56:40.497936 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":280,"threads_started":1}
I20260621 06:56:40.499560 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:40.486470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34029 (local address 127.20.120.193:56845)
0621 06:56:40.487297 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:40.487340 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:40.489611 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:40.490391 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:40.490399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:40.490494 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:40.491124 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:40.491141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:40.492807 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:40.492814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:40.494876 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:40.494889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:40.496035 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:40.496043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:40.496332 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:40.497095 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:56:40.497098 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:40.497145 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:56:40.497263 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:40.497609 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:40.497621 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:40.497629 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 06:56:40.497813 (+ 184us) client_negotiation.cc:240] Negotiation successful
0621 06:56:40.498233 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":238,"threads_started":1}
I20260621 06:56:40.500962 7439 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34029
I20260621 06:56:40.501247 7439 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:40.501988 7439 heartbeater.cc:511] Master 127.20.120.254:34029 requested a full tablet report, sending...
I20260621 06:56:40.503921 7253 ts_manager.cc:194] Registered new tserver with Master: 111ec1795b664573a60df62f233e39f6 (127.20.120.193:35805)
I20260621 06:56:40.505401 7253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.193:56845
W20260621 06:56:40.814313 7443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:40.814564 7443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:40.814646 7443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:40.821565 7443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:40.821656 7443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:40.821759 7443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:56:40.830852 7443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:40.832055 7443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:40.833768 7443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:40.846947 7448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:40.847029 7451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:40.847010 7449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:40.848158 7443 server_base.cc:1034] running on GCE node
I20260621 06:56:40.848832 7443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:40.849795 7443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:40.851042 7443 hybrid_clock.cc:648] HybridClock initialized: now 1782025000850989 us; error 52 us; skew 500 ppm
I20260621 06:56:40.851408 7443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:56:40.853785 7443 webserver.cc:460] Webserver started at http://127.20.120.194:45061/ using document root <none> and password file <none>
I20260621 06:56:40.854555 7443 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:40.854686 7443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:40.855019 7443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:40.858067 7443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "244df1018f39402ea1153549b686e768"
format_stamp: "Formatted at 2026-06-21 06:56:40 on dist-test-slave-gmf2"
server_key: "1a947d2f5d117765c6ae02df36c55eb2"
server_key_iv: "7b7e0abc0cda0131afd579dca1f1e7cb"
server_key_version: "encryptionkey@0"
I20260621 06:56:40.858875 7443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "244df1018f39402ea1153549b686e768"
format_stamp: "Formatted at 2026-06-21 06:56:40 on dist-test-slave-gmf2"
server_key: "1a947d2f5d117765c6ae02df36c55eb2"
server_key_iv: "7b7e0abc0cda0131afd579dca1f1e7cb"
server_key_version: "encryptionkey@0"
I20260621 06:56:40.864339 7443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 06:56:40.867914 7457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:40.869278 7443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 06:56:40.869431 7443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "244df1018f39402ea1153549b686e768"
format_stamp: "Formatted at 2026-06-21 06:56:40 on dist-test-slave-gmf2"
server_key: "1a947d2f5d117765c6ae02df36c55eb2"
server_key_iv: "7b7e0abc0cda0131afd579dca1f1e7cb"
server_key_version: "encryptionkey@0"
I20260621 06:56:40.869608 7443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:40.895776 7443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:40.896633 7443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:40.896899 7443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:40.897828 7443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:40.899309 7443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:40.899394 7443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:40.899468 7443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:40.899528 7443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:40.932482 7443 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:38041
I20260621 06:56:40.932530 7569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:38041 every 8 connection(s)
I20260621 06:56:40.933796 7443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:56:40.942663 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 7443
I20260621 06:56:40.942828 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:56:40.943400 20963 external_mini_cluster.cc:1428] Setting key 30be5705773b5d4fec8428f51cef7498
I20260621 06:56:40.947157 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34029
--builtin_ntp_servers=127.20.120.212:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 06:56:40.947504 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:40.936525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:35657 (local address 127.20.120.254:34029)
0621 06:56:40.936705 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:40.936709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:40.938144 (+ 1435us) server_negotiation.cc:407] Connection header received
0621 06:56:40.939933 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:40.939939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:40.940034 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:40.940146 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:40.941242 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:40.942097 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:40.943900 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:40.944555 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:40.944681 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:40.946499 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:40.946506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:40.946510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:40.946532 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:40.946578 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:40.946594 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:40.946667 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:40.947025 (+ 358us) server_negotiation.cc:299] Negotiation successful
0621 06:56:40.947125 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:56:40.949234 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:40.937063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34029 (local address 127.20.120.194:35657)
0621 06:56:40.937998 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:40.938036 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:40.939666 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:40.940393 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:40.940401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:40.940511 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:40.941070 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:40.941086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:40.942283 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:40.942287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:40.943680 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:40.943692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:40.944937 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:40.944944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:40.945244 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:40.946096 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:56:40.946101 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:40.946158 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:56:40.946256 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:40.946755 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:40.946767 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:40.946775 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 06:56:40.947901 (+ 1126us) client_negotiation.cc:240] Negotiation successful
0621 06:56:40.948235 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":264,"threads_started":1}
I20260621 06:56:40.951300 7570 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34029
I20260621 06:56:40.951638 7570 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:40.952411 7570 heartbeater.cc:511] Master 127.20.120.254:34029 requested a full tablet report, sending...
I20260621 06:56:40.953893 7253 ts_manager.cc:194] Registered new tserver with Master: 244df1018f39402ea1153549b686e768 (127.20.120.194:38041)
I20260621 06:56:40.954803 7253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.194:35657
W20260621 06:56:41.282884 7573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:41.283145 7573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:41.283231 7573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:41.290340 7573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:41.290432 7573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:41.290581 7573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:56:41.299446 7573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:34029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:41.300606 7573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:41.304284 7573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:41.317674 7579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:41.317749 7578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:41.317723 7581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:41.318691 7573 server_base.cc:1034] running on GCE node
I20260621 06:56:41.319327 7573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:41.320274 7573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:41.321516 7573 hybrid_clock.cc:648] HybridClock initialized: now 1782025001321464 us; error 63 us; skew 500 ppm
I20260621 06:56:41.321849 7573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:56:41.324168 7573 webserver.cc:460] Webserver started at http://127.20.120.195:42573/ using document root <none> and password file <none>
I20260621 06:56:41.324872 7573 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:41.324999 7573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:41.325299 7573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:41.328159 7573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "9c8417c0fee549f695d1dd8802a99c4e"
format_stamp: "Formatted at 2026-06-21 06:56:41 on dist-test-slave-gmf2"
server_key: "1bad940acd6f9c7cbd4c672f248fce81"
server_key_iv: "3635b283e4308729c506a63e80204665"
server_key_version: "encryptionkey@0"
I20260621 06:56:41.328945 7573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "9c8417c0fee549f695d1dd8802a99c4e"
format_stamp: "Formatted at 2026-06-21 06:56:41 on dist-test-slave-gmf2"
server_key: "1bad940acd6f9c7cbd4c672f248fce81"
server_key_iv: "3635b283e4308729c506a63e80204665"
server_key_version: "encryptionkey@0"
I20260621 06:56:41.334092 7573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260621 06:56:41.337387 7587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:41.338707 7573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:56:41.338886 7573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "9c8417c0fee549f695d1dd8802a99c4e"
format_stamp: "Formatted at 2026-06-21 06:56:41 on dist-test-slave-gmf2"
server_key: "1bad940acd6f9c7cbd4c672f248fce81"
server_key_iv: "3635b283e4308729c506a63e80204665"
server_key_version: "encryptionkey@0"
I20260621 06:56:41.339095 7573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:41.380504 7573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:41.381382 7573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:41.381708 7573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:41.382908 7573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:41.384447 7573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:41.384593 7573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:41.384727 7573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:41.384820 7573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:41.416642 7573 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:37925
I20260621 06:56:41.416693 7699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:37925 every 8 connection(s)
I20260621 06:56:41.418081 7573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:56:41.424763 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 7573
I20260621 06:56:41.424924 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:56:41.425457 20963 external_mini_cluster.cc:1428] Setting key 3187be20e745b65697664d050ea5e4ab
I20260621 06:56:41.430490 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.420589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:51183 (local address 127.20.120.254:34029)
0621 06:56:41.420786 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:41.420791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:41.422143 (+ 1352us) server_negotiation.cc:407] Connection header received
0621 06:56:41.423943 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:41.423948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:41.424012 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:41.424096 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:41.425224 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.426217 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.427923 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.428350 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.428486 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.429671 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:41.429692 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:41.429695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:41.429711 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:41.429743 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:41.429754 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:41.429847 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:41.430094 (+ 247us) server_negotiation.cc:299] Negotiation successful
0621 06:56:41.430184 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 06:56:41.431438 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.421071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:34029 (local address 127.20.120.195:51183)
0621 06:56:41.421984 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:41.422036 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:41.423702 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:41.424280 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:41.424287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:41.424387 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:41.425062 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:41.425080 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.426392 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.426405 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:41.427765 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:41.427777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.428569 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.428576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:41.428784 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.429378 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:56:41.429380 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:41.429427 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:56:41.429498 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:41.429845 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:41.429855 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:41.429865 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 06:56:41.430085 (+ 220us) client_negotiation.cc:240] Negotiation successful
0621 06:56:41.430457 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":251,"threads_started":1}
I20260621 06:56:41.432618 7700 heartbeater.cc:347] Connected to a master server at 127.20.120.254:34029
I20260621 06:56:41.432875 7700 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:41.433563 7700 heartbeater.cc:511] Master 127.20.120.254:34029 requested a full tablet report, sending...
I20260621 06:56:41.435053 7253 ts_manager.cc:194] Registered new tserver with Master: 9c8417c0fee549f695d1dd8802a99c4e (127.20.120.195:37925)
I20260621 06:56:41.437700 7253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.195:51183
I20260621 06:56:41.441004 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:56:41.451372 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.445636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.20.120.254:34029)
0621 06:56:41.445895 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:41.445920 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:41.446327 (+ 407us) server_negotiation.cc:407] Connection header received
0621 06:56:41.446494 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:41.446497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:41.446566 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:41.446644 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:41.447085 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.447749 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.449077 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.449435 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.449530 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.450667 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:41.450671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:41.450674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:41.450690 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:41.450721 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:41.450732 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:41.450837 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:41.451003 (+ 166us) server_negotiation.cc:299] Negotiation successful
0621 06:56:41.451078 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 06:56:41.455374 7253 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:56:41.463194 7253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:56:41.498039 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.476161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:35805 (local address 127.0.0.1:60690)
0621 06:56:41.476909 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:41.476949 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:41.477142 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:41.478426 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:41.478437 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:41.478479 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:41.479040 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:41.479056 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.492198 (+ 13142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.492206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:41.494836 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:41.494849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.497051 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.497062 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:41.497267 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.497317 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 06:56:41.497463 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 06:56:41.497657 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":295,"threads_started":1}
I20260621 06:56:41.498129 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.476513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.20.120.193:35805)
0621 06:56:41.477026 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:41.477034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:41.477064 (+ 30us) server_negotiation.cc:407] Connection header received
0621 06:56:41.477463 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:41.477472 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:41.477741 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:41.478075 (+ 334us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:41.479286 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.491993 (+ 12707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.495042 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.496511 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.496698 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.496904 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:41.497535 (+ 631us) server_negotiation.cc:299] Negotiation successful
0621 06:56:41.497719 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":684,"threads_started":1}
I20260621 06:56:41.499006 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.476903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:38041 (local address 127.0.0.1:59674)
0621 06:56:41.487981 (+ 11078us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:41.488006 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:41.488154 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:41.490001 (+ 1847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:41.490010 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:41.490045 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:41.490508 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:41.490523 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.493466 (+ 2943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.493474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:41.495657 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:41.495671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.498254 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.498263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:41.498472 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.498515 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 06:56:41.498590 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 06:56:41.498739 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10888,"thread_start_us":11349,"threads_started":1}
I20260621 06:56:41.499343 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.477650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.20.120.194:38041)
0621 06:56:41.488935 (+ 11285us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:41.488961 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:41.489001 (+ 40us) server_negotiation.cc:407] Connection header received
0621 06:56:41.489129 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:41.489140 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:41.489433 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:41.489654 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:41.490754 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.493311 (+ 2557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.495865 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.497583 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.497696 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.497922 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:41.498803 (+ 881us) server_negotiation.cc:299] Negotiation successful
0621 06:56:41.498976 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11137,"thread_start_us":11021,"threads_started":1}
I20260621 06:56:41.502063 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.477255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:37925 (local address 127.0.0.1:38210)
0621 06:56:41.492478 (+ 15223us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:41.492505 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:41.492607 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:41.494512 (+ 1905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:41.494516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:41.494550 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:41.495129 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:41.495138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.496783 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.496789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:41.498723 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:41.498746 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.501310 (+ 2564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.501317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:41.501524 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.501555 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 06:56:41.501641 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 06:56:41.501762 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15099,"thread_start_us":14949,"threads_started":1}
I20260621 06:56:41.502760 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.477566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.20.120.195:37925)
0621 06:56:41.493935 (+ 16369us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:41.493947 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:41.493994 (+ 47us) server_negotiation.cc:407] Connection header received
0621 06:56:41.494088 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:41.494096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:41.494320 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:41.494508 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:41.495290 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.496651 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.498952 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.500783 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.500973 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.501188 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:41.502052 (+ 864us) server_negotiation.cc:299] Negotiation successful
0621 06:56:41.502406 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16167,"thread_start_us":16013,"threads_started":1}
I20260621 06:56:41.502522 7374 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc (DEFAULT_TABLE table=test-table [id=3568ca81502f403fbc30e5096f141069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:41.502532 7505 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc (DEFAULT_TABLE table=test-table [id=3568ca81502f403fbc30e5096f141069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:41.503924 7374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:41.503963 7505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:41.506808 7635 tablet_service.cc:1467] Processing CreateTablet for tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc (DEFAULT_TABLE table=test-table [id=3568ca81502f403fbc30e5096f141069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:41.508369 7635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:41.508471 7439 heartbeater.cc:503] Master 127.20.120.254:34029 was elected leader, sending a full tablet report...
I20260621 06:56:41.518860 7718 tablet_bootstrap.cc:492] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6: Bootstrap starting.
I20260621 06:56:41.522082 7718 tablet_bootstrap.cc:654] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:41.522671 7719 tablet_bootstrap.cc:492] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768: Bootstrap starting.
I20260621 06:56:41.523653 7718 log.cc:824] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:41.524202 7720 tablet_bootstrap.cc:492] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e: Bootstrap starting.
I20260621 06:56:41.526575 7719 tablet_bootstrap.cc:654] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:41.527722 7720 tablet_bootstrap.cc:654] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:41.527875 7719 log.cc:824] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:41.528523 7718 tablet_bootstrap.cc:492] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6: No bootstrap required, opened a new log
I20260621 06:56:41.528815 7718 ts_tablet_manager.cc:1397] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260621 06:56:41.528927 7720 log.cc:824] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:41.531272 7719 tablet_bootstrap.cc:492] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768: No bootstrap required, opened a new log
I20260621 06:56:41.531545 7719 ts_tablet_manager.cc:1397] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 06:56:41.531574 7720 tablet_bootstrap.cc:492] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e: No bootstrap required, opened a new log
I20260621 06:56:41.531816 7720 ts_tablet_manager.cc:1397] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 06:56:41.539584 7720 raft_consensus.cc:348] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111ec1795b664573a60df62f233e39f6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35805 } } peers { permanent_uuid: "9c8417c0fee549f695d1dd8802a99c4e" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37925 } } peers { permanent_uuid: "244df1018f39402ea1153549b686e768" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38041 } }
I20260621 06:56:41.539695 7718 raft_consensus.cc:348] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111ec1795b664573a60df62f233e39f6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35805 } } peers { permanent_uuid: "9c8417c0fee549f695d1dd8802a99c4e" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37925 } } peers { permanent_uuid: "244df1018f39402ea1153549b686e768" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38041 } }
I20260621 06:56:41.540001 7720 raft_consensus.cc:374] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:41.540093 7720 raft_consensus.cc:729] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c8417c0fee549f695d1dd8802a99c4e, State: Initialized, Role: FOLLOWER
I20260621 06:56:41.540132 7718 raft_consensus.cc:374] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:41.540220 7718 raft_consensus.cc:729] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111ec1795b664573a60df62f233e39f6, State: Initialized, Role: FOLLOWER
I20260621 06:56:41.540772 7720 consensus_queue.cc:260] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111ec1795b664573a60df62f233e39f6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35805 } } peers { permanent_uuid: "9c8417c0fee549f695d1dd8802a99c4e" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37925 } } peers { permanent_uuid: "244df1018f39402ea1153549b686e768" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38041 } }
I20260621 06:56:41.540920 7718 consensus_queue.cc:260] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111ec1795b664573a60df62f233e39f6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35805 } } peers { permanent_uuid: "9c8417c0fee549f695d1dd8802a99c4e" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37925 } } peers { permanent_uuid: "244df1018f39402ea1153549b686e768" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38041 } }
I20260621 06:56:41.541956 7700 heartbeater.cc:503] Master 127.20.120.254:34029 was elected leader, sending a full tablet report...
I20260621 06:56:41.542256 7719 raft_consensus.cc:348] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111ec1795b664573a60df62f233e39f6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35805 } } peers { permanent_uuid: "9c8417c0fee549f695d1dd8802a99c4e" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37925 } } peers { permanent_uuid: "244df1018f39402ea1153549b686e768" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38041 } }
I20260621 06:56:41.542714 7719 raft_consensus.cc:374] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:41.542816 7719 raft_consensus.cc:729] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244df1018f39402ea1153549b686e768, State: Initialized, Role: FOLLOWER
I20260621 06:56:41.543569 7720 ts_tablet_manager.cc:1428] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260621 06:56:41.543572 7719 consensus_queue.cc:260] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111ec1795b664573a60df62f233e39f6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35805 } } peers { permanent_uuid: "9c8417c0fee549f695d1dd8802a99c4e" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37925 } } peers { permanent_uuid: "244df1018f39402ea1153549b686e768" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38041 } }
I20260621 06:56:41.544859 7718 ts_tablet_manager.cc:1428] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260621 06:56:41.545236 7725 raft_consensus.cc:482] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:56:41.545522 7725 raft_consensus.cc:504] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111ec1795b664573a60df62f233e39f6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35805 } } peers { permanent_uuid: "9c8417c0fee549f695d1dd8802a99c4e" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37925 } } peers { permanent_uuid: "244df1018f39402ea1153549b686e768" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38041 } }
I20260621 06:56:41.546703 7570 heartbeater.cc:503] Master 127.20.120.254:34029 was elected leader, sending a full tablet report...
I20260621 06:56:41.547585 7719 ts_tablet_manager.cc:1428] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260621 06:56:41.549139 7725 leader_election.cc:288] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c8417c0fee549f695d1dd8802a99c4e (127.20.120.195:37925), 244df1018f39402ea1153549b686e768 (127.20.120.194:38041)
I20260621 06:56:41.560554 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.549711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:59425 (local address 127.20.120.195:37925)
0621 06:56:41.549967 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:41.549977 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:41.554612 (+ 4635us) server_negotiation.cc:407] Connection header received
0621 06:56:41.554740 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:41.554745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:41.554816 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:41.554904 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:41.555516 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.556291 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.558509 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.559685 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.559821 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.559890 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:41.560187 (+ 297us) server_negotiation.cc:299] Negotiation successful
0621 06:56:41.560313 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 06:56:41.560554 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.550288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:46407 (local address 127.20.120.194:38041)
0621 06:56:41.550513 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:41.550522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:41.554742 (+ 4220us) server_negotiation.cc:407] Connection header received
0621 06:56:41.554941 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:41.554946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:41.555028 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:41.555129 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:41.555713 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.556390 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.558388 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.559596 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.559765 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.559835 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:41.560188 (+ 353us) server_negotiation.cc:299] Negotiation successful
0621 06:56:41.560313 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 06:56:41.560752 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.549584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:38041 (local address 127.20.120.193:46407)
0621 06:56:41.554659 (+ 5075us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:41.554675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:41.554793 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:41.555154 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:41.555157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:41.555175 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:41.555545 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:41.555559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.556522 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.556526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:41.558205 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:41.558216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.559800 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.559809 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:41.560016 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.560052 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 06:56:41.560148 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 06:56:41.560341 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4939,"mutex_wait_us":155,"thread_start_us":4547,"threads_started":1}
I20260621 06:56:41.560770 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.549584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:37925 (local address 127.20.120.193:59425)
0621 06:56:41.554509 (+ 4925us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:41.554525 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:41.554633 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:41.554933 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:41.554937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:41.554963 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:41.555353 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:41.555366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.556416 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.556423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:41.558244 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:41.558261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.559870 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.559877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:41.560048 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.560071 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 06:56:41.560148 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 06:56:41.560335 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4801,"mutex_wait_us":139,"thread_start_us":4615,"threads_started":1}
I20260621 06:56:41.561683 7655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8e2675c0d24f5d8534c1a0eda1b4fc" candidate_uuid: "111ec1795b664573a60df62f233e39f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c8417c0fee549f695d1dd8802a99c4e" is_pre_election: true
I20260621 06:56:41.561785 7525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8e2675c0d24f5d8534c1a0eda1b4fc" candidate_uuid: "111ec1795b664573a60df62f233e39f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244df1018f39402ea1153549b686e768" is_pre_election: true
I20260621 06:56:41.562279 7655 raft_consensus.cc:2454] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 111ec1795b664573a60df62f233e39f6 in term 0.
I20260621 06:56:41.562279 7525 raft_consensus.cc:2454] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 111ec1795b664573a60df62f233e39f6 in term 0.
I20260621 06:56:41.562933 7327 leader_election.cc:302] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111ec1795b664573a60df62f233e39f6, 9c8417c0fee549f695d1dd8802a99c4e; no voters:
I20260621 06:56:41.563288 7725 raft_consensus.cc:2781] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:56:41.563398 7725 raft_consensus.cc:482] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:56:41.563486 7725 raft_consensus.cc:3037] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:41.565476 7725 raft_consensus.cc:504] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111ec1795b664573a60df62f233e39f6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35805 } } peers { permanent_uuid: "9c8417c0fee549f695d1dd8802a99c4e" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37925 } } peers { permanent_uuid: "244df1018f39402ea1153549b686e768" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38041 } }
I20260621 06:56:41.566064 7725 leader_election.cc:288] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [CANDIDATE]: Term 1 election: Requested vote from peers 9c8417c0fee549f695d1dd8802a99c4e (127.20.120.195:37925), 244df1018f39402ea1153549b686e768 (127.20.120.194:38041)
I20260621 06:56:41.566437 7655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8e2675c0d24f5d8534c1a0eda1b4fc" candidate_uuid: "111ec1795b664573a60df62f233e39f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c8417c0fee549f695d1dd8802a99c4e"
I20260621 06:56:41.566547 7525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d8e2675c0d24f5d8534c1a0eda1b4fc" candidate_uuid: "111ec1795b664573a60df62f233e39f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244df1018f39402ea1153549b686e768"
I20260621 06:56:41.566645 7655 raft_consensus.cc:3037] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:41.566730 7525 raft_consensus.cc:3037] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:41.569819 7655 raft_consensus.cc:2454] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 111ec1795b664573a60df62f233e39f6 in term 1.
I20260621 06:56:41.569819 7525 raft_consensus.cc:2454] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 111ec1795b664573a60df62f233e39f6 in term 1.
I20260621 06:56:41.570199 7327 leader_election.cc:302] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111ec1795b664573a60df62f233e39f6, 9c8417c0fee549f695d1dd8802a99c4e; no voters:
I20260621 06:56:41.570485 7725 raft_consensus.cc:2781] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:41.570895 7725 raft_consensus.cc:686] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 1 LEADER]: Becoming Leader. State: Replica: 111ec1795b664573a60df62f233e39f6, State: Running, Role: LEADER
I20260621 06:56:41.571350 7725 consensus_queue.cc:237] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111ec1795b664573a60df62f233e39f6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35805 } } peers { permanent_uuid: "9c8417c0fee549f695d1dd8802a99c4e" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37925 } } peers { permanent_uuid: "244df1018f39402ea1153549b686e768" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38041 } }
I20260621 06:56:41.575901 7251 catalog_manager.cc:5495] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 111ec1795b664573a60df62f233e39f6 (127.20.120.193). New cstate: current_term: 1 leader_uuid: "111ec1795b664573a60df62f233e39f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111ec1795b664573a60df62f233e39f6" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 35805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c8417c0fee549f695d1dd8802a99c4e" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "244df1018f39402ea1153549b686e768" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38041 } health_report { overall_health: UNKNOWN } } }
I20260621 06:56:41.605259 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.600578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.20.120.193:35805)
0621 06:56:41.600748 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:41.600753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:41.600769 (+ 16us) server_negotiation.cc:407] Connection header received
0621 06:56:41.600879 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:41.600883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:41.600970 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:41.601084 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:56:41.601409 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.602148 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.603432 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.603840 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.603926 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.604355 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:56:41.604807 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:56:41.604928 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:41.605041 (+ 113us) server_negotiation.cc:299] Negotiation successful
0621 06:56:41.605075 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 06:56:41.613344 7525 raft_consensus.cc:1261] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768 [term 1 FOLLOWER]: Refusing update from remote peer 111ec1795b664573a60df62f233e39f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:56:41.613466 7655 raft_consensus.cc:1261] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e [term 1 FOLLOWER]: Refusing update from remote peer 111ec1795b664573a60df62f233e39f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:56:41.614362 7729 consensus_queue.cc:1035] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244df1018f39402ea1153549b686e768" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 38041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:56:41.614706 7725 consensus_queue.cc:1035] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c8417c0fee549f695d1dd8802a99c4e" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 37925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:56:41.642131 7251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:56:41.642429 7251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:56:41.646199 7251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 515995b1048a40c28f568b1e3d17e8fa: Sending DeleteTablet for 3 replicas of tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc
I20260621 06:56:41.647343 7374 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:41 UTC) from {username='slave'} at 127.0.0.1:60690
I20260621 06:56:41.647378 7635 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:41 UTC) from {username='slave'} at 127.0.0.1:38210
I20260621 06:56:41.647576 7505 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:41 UTC) from {username='slave'} at 127.0.0.1:59674
I20260621 06:56:41.648759 7743 tablet_replica.cc:330] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6: stopping tablet replica
I20260621 06:56:41.649183 7743 raft_consensus.cc:2229] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:56:41.649847 7743 raft_consensus.cc:2258] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:41.652148 7743 ts_tablet_manager.cc:1905] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:41.652524 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 7312
I20260621 06:56:41.659888 7743 ts_tablet_manager.cc:1918] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:41.660091 7743 log.cc:1196] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-0/wal/wals/5d8e2675c0d24f5d8534c1a0eda1b4fc
I20260621 06:56:41.660678 7743 ts_tablet_manager.cc:1939] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 111ec1795b664573a60df62f233e39f6: Deleting consensus metadata
I20260621 06:56:41.662482 7742 tablet_replica.cc:330] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e: stopping tablet replica
I20260621 06:56:41.662961 7742 raft_consensus.cc:2229] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:56:41.663550 7742 raft_consensus.cc:2258] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:41.665688 7742 ts_tablet_manager.cc:1905] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:41.668730 7744 tablet_replica.cc:330] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768: stopping tablet replica
I20260621 06:56:41.669117 7744 raft_consensus.cc:2229] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:56:41.669584 7744 raft_consensus.cc:2258] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:41.671852 7744 ts_tablet_manager.cc:1905] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:41.672894 7742 ts_tablet_manager.cc:1918] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:41.673063 7742 log.cc:1196] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/5d8e2675c0d24f5d8534c1a0eda1b4fc
I20260621 06:56:41.674230 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.653716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.20.120.193:35805)
0621 06:56:41.653971 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:41.653984 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:41.662729 (+ 8745us) server_negotiation.cc:407] Connection header received
0621 06:56:41.663059 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:41.663069 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:41.663176 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:41.663283 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:41.664152 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.665733 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.670016 (+ 4283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.670709 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.670821 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.672430 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:41.672443 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:41.672472 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:41.673259 (+ 787us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:41.673316 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:41.673343 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:41.673420 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:41.673718 (+ 298us) server_negotiation.cc:299] Negotiation successful
0621 06:56:41.673830 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:56:41.674185 7237 catalog_manager.cc:4841] TS 111ec1795b664573a60df62f233e39f6 (127.20.120.193:35805): tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc (table test-table [id=3568ca81502f403fbc30e5096f141069]) successfully deleted
I20260621 06:56:41.675037 7742 ts_tablet_manager.cc:1939] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 9c8417c0fee549f695d1dd8802a99c4e: Deleting consensus metadata
I20260621 06:56:41.675282 7434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:41.677238 7237 catalog_manager.cc:4841] TS 9c8417c0fee549f695d1dd8802a99c4e (127.20.120.195:37925): tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc (table test-table [id=3568ca81502f403fbc30e5096f141069]) successfully deleted
I20260621 06:56:41.680330 7744 ts_tablet_manager.cc:1918] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:41.680495 7744 log.cc:1196] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/5d8e2675c0d24f5d8534c1a0eda1b4fc
I20260621 06:56:41.680925 7744 ts_tablet_manager.cc:1939] T 5d8e2675c0d24f5d8534c1a0eda1b4fc P 244df1018f39402ea1153549b686e768: Deleting consensus metadata
I20260621 06:56:41.682412 7238 catalog_manager.cc:4841] TS 244df1018f39402ea1153549b686e768 (127.20.120.194:38041): tablet 5d8e2675c0d24f5d8534c1a0eda1b4fc (table test-table [id=3568ca81502f403fbc30e5096f141069]) successfully deleted
I20260621 06:56:41.774214 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 7312
I20260621 06:56:41.794159 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 7443
I20260621 06:56:41.801836 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.795102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.20.120.194:38041)
0621 06:56:41.795291 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:41.795295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:41.795313 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:56:41.795380 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:41.795384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:41.795478 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:41.795589 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:41.796039 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.796819 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.798230 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.798679 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.798899 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.800436 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:41.800446 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:41.800462 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:41.801083 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:41.801131 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:41.801152 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:41.801251 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:41.801446 (+ 195us) server_negotiation.cc:299] Negotiation successful
0621 06:56:41.801559 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 06:56:41.802742 7565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:41.892417 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 7443
I20260621 06:56:41.909971 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 7573
I20260621 06:56:41.917121 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:41.910813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.20.120.195:37925)
0621 06:56:41.911075 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:41.911081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:41.911102 (+ 21us) server_negotiation.cc:407] Connection header received
0621 06:56:41.911167 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:41.911180 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:41.911270 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:41.911394 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:41.911797 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.912581 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.914043 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:41.914485 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:41.914630 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:41.915726 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:41.915743 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:41.915759 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:41.916354 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:41.916401 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:41.916421 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:41.916550 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:41.916683 (+ 133us) server_negotiation.cc:299] Negotiation successful
0621 06:56:41.916785 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260621 06:56:41.917793 7695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:42.025346 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 7573
I20260621 06:56:42.044056 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 7222
I20260621 06:56:42.045120 7283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:42.153522 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 7222
2026-06-21T06:56:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2699 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:56:42 dist-test-slave-gmf2 krb5kdc[7752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:56:42 dist-test-slave-gmf2 krb5kdc[7752](info): set up 2 sockets
Jun 21 06:56:42 dist-test-slave-gmf2 krb5kdc[7752](info): commencing operation
krb5kdc: starting...
W20260621 06:56:43.254151 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:56:43 dist-test-slave-gmf2 krb5kdc[7752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:56:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:56:43.481331 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:46361
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:46361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:43.795436 7765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:43.795688 7765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:43.795773 7765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:43.802639 7765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:56:43.802752 7765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:43.802795 7765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:43.802888 7765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:56:43.802933 7765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:56:43.812135 7765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:46361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:46361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:43.813342 7765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:43.814637 7765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:43.826584 7770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:43.826649 7773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:43.826614 7771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:43.827246 7765 server_base.cc:1034] running on GCE node
I20260621 06:56:43.828050 7765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:43.829478 7765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:43.830669 7765 hybrid_clock.cc:648] HybridClock initialized: now 1782025003830627 us; error 43 us; skew 500 ppm
I20260621 06:56:43.831002 7765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:43 dist-test-slave-gmf2 krb5kdc[7752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025003, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:43.837632 7765 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:56:43.839210 7765 webserver.cc:460] Webserver started at http://127.20.120.254:38989/ using document root <none> and password file <none>
I20260621 06:56:43.839843 7765 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:43.839946 7765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:43.840188 7765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:43.843246 7765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "d9951e3f8ee34e6aaf16b0342024d825"
format_stamp: "Formatted at 2026-06-21 06:56:43 on dist-test-slave-gmf2"
server_key: "1c448914299917bb3de17638f8e1c4c8"
server_key_iv: "cdb6ad179ba265a0ba60beaf31ea3b14"
server_key_version: "encryptionkey@0"
I20260621 06:56:43.843935 7765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "d9951e3f8ee34e6aaf16b0342024d825"
format_stamp: "Formatted at 2026-06-21 06:56:43 on dist-test-slave-gmf2"
server_key: "1c448914299917bb3de17638f8e1c4c8"
server_key_iv: "cdb6ad179ba265a0ba60beaf31ea3b14"
server_key_version: "encryptionkey@0"
I20260621 06:56:43.849030 7765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 06:56:43.852361 7780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:43.853751 7765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 06:56:43.854007 7765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "d9951e3f8ee34e6aaf16b0342024d825"
format_stamp: "Formatted at 2026-06-21 06:56:43 on dist-test-slave-gmf2"
server_key: "1c448914299917bb3de17638f8e1c4c8"
server_key_iv: "cdb6ad179ba265a0ba60beaf31ea3b14"
server_key_version: "encryptionkey@0"
I20260621 06:56:43.854240 7765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:43.867944 7765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:43.871949 7765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:43.872231 7765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:43.890417 7765 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:46361
I20260621 06:56:43.890429 7831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:46361 every 8 connection(s)
I20260621 06:56:43.891822 7765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:56:43.895325 7832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:43.901603 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 7765
I20260621 06:56:43.901825 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:56:43.902287 20963 external_mini_cluster.cc:1428] Setting key 366ea33e03b33d9117cb5c12d2cbeee2
I20260621 06:56:43.905594 7832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825: Bootstrap starting.
I20260621 06:56:43.909135 7832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:43.910209 7832 log.cc:824] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:43.912715 7832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825: No bootstrap required, opened a new log
Jun 21 06:56:43 dist-test-slave-gmf2 krb5kdc[7752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:56:43.920320 7832 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9951e3f8ee34e6aaf16b0342024d825" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46361 } }
I20260621 06:56:43.920718 7832 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:43.920821 7832 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9951e3f8ee34e6aaf16b0342024d825, State: Initialized, Role: FOLLOWER
I20260621 06:56:43.921514 7832 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9951e3f8ee34e6aaf16b0342024d825" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46361 } }
I20260621 06:56:43.921813 7832 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:56:43.921968 7832 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:56:43.922154 7832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:43.924431 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:43.904048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.20.120.254:46361)
0621 06:56:43.904798 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:43.904809 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:43.904848 (+ 39us) server_negotiation.cc:407] Connection header received
0621 06:56:43.906600 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:43.906621 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:43.907010 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:43.907326 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:43.909146 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:43.910655 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:43.911995 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:43.912629 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:43.912816 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:43.916587 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:43.916594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:43.916609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:43.916648 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:43.919618 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:43.920311 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:43.920317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:43.920323 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:43.920477 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:43.920946 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:43.920953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:43.920956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:43.921615 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:43.921983 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:43.922700 (+ 717us) server_negotiation.cc:299] Negotiation successful
0621 06:56:43.923141 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":361,"threads_started":1}
I20260621 06:56:43.925328 7832 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9951e3f8ee34e6aaf16b0342024d825" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46361 } }
I20260621 06:56:43.925863 7832 leader_election.cc:302] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9951e3f8ee34e6aaf16b0342024d825; no voters:
I20260621 06:56:43.926499 7832 leader_election.cc:288] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:56:43.926631 7837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:43.928094 7837 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [term 1 LEADER]: Becoming Leader. State: Replica: d9951e3f8ee34e6aaf16b0342024d825, State: Running, Role: LEADER
I20260621 06:56:43.928610 7832 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:56:43.928591 7837 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9951e3f8ee34e6aaf16b0342024d825" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46361 } }
I20260621 06:56:43.931488 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9951e3f8ee34e6aaf16b0342024d825. Latest consensus state: current_term: 1 leader_uuid: "d9951e3f8ee34e6aaf16b0342024d825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9951e3f8ee34e6aaf16b0342024d825" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46361 } } }
I20260621 06:56:43.931931 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:43.932252 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9951e3f8ee34e6aaf16b0342024d825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9951e3f8ee34e6aaf16b0342024d825" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 46361 } } }
I20260621 06:56:43.932564 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:43.938932 7846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:56:43.943351 7846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:56:43.951995 7846 catalog_manager.cc:1347] Generated new cluster ID: 1f1c773ecfe744da8010ba000fcdb11b
I20260621 06:56:43.952114 7846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:56:43.973603 7846 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:56:43.974978 7846 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:56:43.981228 7846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825: Generated new TSK 0
I20260621 06:56:43.982019 7846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.120.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
Entry for principal kudu/127.20.120.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab.
I20260621 06:56:44.033439 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:46361
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:44.348737 7858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:44.348995 7858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:44.349089 7858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:44.356093 7858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:44.356190 7858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:44.356304 7858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:56:44.365557 7858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.193.keytab
--principal=kudu/127.20.120.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:46361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:44.366712 7858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:44.367877 7858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:44.380609 7863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:44.380666 7864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:44.380712 7866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:44.381757 7858 server_base.cc:1034] running on GCE node
I20260621 06:56:44.382350 7858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:44.383211 7858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:44.384439 7858 hybrid_clock.cc:648] HybridClock initialized: now 1782025004384393 us; error 56 us; skew 500 ppm
I20260621 06:56:44.384732 7858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:44 dist-test-slave-gmf2 krb5kdc[7752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025004, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:44.387714 7858 init.cc:377] Logged in from keytab as kudu/127.20.120.193@KRBTEST.COM (short username kudu)
I20260621 06:56:44.389038 7858 webserver.cc:460] Webserver started at http://127.20.120.193:38813/ using document root <none> and password file <none>
I20260621 06:56:44.389667 7858 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:44.389809 7858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:44.390159 7858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:44.392941 7858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "e306100a914342babf22933a23b86292"
format_stamp: "Formatted at 2026-06-21 06:56:44 on dist-test-slave-gmf2"
server_key: "1fb7cc7529a644f513625ad02d17be55"
server_key_iv: "ae16a9f8c1cc3c36eb647b5f729bb831"
server_key_version: "encryptionkey@0"
I20260621 06:56:44.393709 7858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "e306100a914342babf22933a23b86292"
format_stamp: "Formatted at 2026-06-21 06:56:44 on dist-test-slave-gmf2"
server_key: "1fb7cc7529a644f513625ad02d17be55"
server_key_iv: "ae16a9f8c1cc3c36eb647b5f729bb831"
server_key_version: "encryptionkey@0"
I20260621 06:56:44.398612 7858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 06:56:44.401782 7873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:44.403123 7858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:56:44.403301 7858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "e306100a914342babf22933a23b86292"
format_stamp: "Formatted at 2026-06-21 06:56:44 on dist-test-slave-gmf2"
server_key: "1fb7cc7529a644f513625ad02d17be55"
server_key_iv: "ae16a9f8c1cc3c36eb647b5f729bb831"
server_key_version: "encryptionkey@0"
I20260621 06:56:44.403498 7858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:44.418017 7858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:44.422021 7858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:44.422307 7858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:44.423386 7858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:44.425009 7858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:44.425119 7858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:44.425247 7858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:44.425330 7858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:44.457520 7858 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:34631
I20260621 06:56:44.457568 7985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:34631 every 8 connection(s)
I20260621 06:56:44.458815 7858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:56:44.463752 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 7858
I20260621 06:56:44.463946 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:56:44.464515 20963 external_mini_cluster.cc:1428] Setting key 359de65f038c6edf394870fa073d947f
Jun 21 06:56:44 dist-test-slave-gmf2 krb5kdc[7752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025004, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.193@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:56:44.480098 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:44.461019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:56977 (local address 127.20.120.254:46361)
0621 06:56:44.461467 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:44.461471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:44.462317 (+ 846us) server_negotiation.cc:407] Connection header received
0621 06:56:44.464414 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:44.464419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:44.464488 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:44.464607 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:44.466387 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:44.467068 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:44.469466 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:44.469838 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:44.469992 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:44.474144 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:44.474148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:44.474151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:44.474181 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:44.477220 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:44.477831 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:44.477839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:44.477844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:44.477997 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:44.478371 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:44.478378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:44.478384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:44.478746 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:44.478880 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:44.479434 (+ 554us) server_negotiation.cc:299] Negotiation successful
0621 06:56:44.479663 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":244,"threads_started":1}
I20260621 06:56:44.480702 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:44.461443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:46361 (local address 127.20.120.193:56977)
0621 06:56:44.462165 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:44.462203 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:44.464106 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:44.464854 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:44.464865 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:44.465516 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:44.466196 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:44.466250 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:44.467227 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:44.467234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:44.469331 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:44.469347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:44.470060 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:44.470068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:44.470379 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:44.471061 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:56:44.471092 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:44.473931 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:44.477350 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:44.477355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:44.477365 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:44.477713 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:44.478096 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:44.478099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:44.478101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:44.478257 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:44.478862 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:44.478867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:44.479114 (+ 247us) client_negotiation.cc:769] Sending connection context
0621 06:56:44.479293 (+ 179us) client_negotiation.cc:240] Negotiation successful
0621 06:56:44.479618 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":253,"threads_started":1}
I20260621 06:56:44.482380 7986 heartbeater.cc:347] Connected to a master server at 127.20.120.254:46361
I20260621 06:56:44.482666 7986 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:44.483387 7986 heartbeater.cc:511] Master 127.20.120.254:46361 requested a full tablet report, sending...
I20260621 06:56:44.485435 7797 ts_manager.cc:194] Registered new tserver with Master: e306100a914342babf22933a23b86292 (127.20.120.193:34631)
I20260621 06:56:44.487461 7797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.193@KRBTEST.COM'} at 127.20.120.193:56977
WARNING: no policy specified for kudu/127.20.120.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
Entry for principal kudu/127.20.120.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab.
I20260621 06:56:44.520738 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:46361
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:44.829476 7992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:44.829730 7992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:44.829818 7992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:44.836786 7992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:44.836895 7992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:44.837028 7992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:56:44.846513 7992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.194.keytab
--principal=kudu/127.20.120.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:46361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:44.847739 7992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:44.848989 7992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:44.862206 7997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:44.862249 7998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:44.862287 8000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:44.863037 7992 server_base.cc:1034] running on GCE node
I20260621 06:56:44.863620 7992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:44.864571 7992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:44.865798 7992 hybrid_clock.cc:648] HybridClock initialized: now 1782025004865762 us; error 38 us; skew 500 ppm
I20260621 06:56:44.866164 7992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:44 dist-test-slave-gmf2 krb5kdc[7752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025004, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:44.869386 7992 init.cc:377] Logged in from keytab as kudu/127.20.120.194@KRBTEST.COM (short username kudu)
I20260621 06:56:44.870955 7992 webserver.cc:460] Webserver started at http://127.20.120.194:36653/ using document root <none> and password file <none>
I20260621 06:56:44.871660 7992 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:44.871809 7992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:44.872123 7992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:44.874974 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "b71d080af9c3449eb2760e39b3e52150"
format_stamp: "Formatted at 2026-06-21 06:56:44 on dist-test-slave-gmf2"
server_key: "0b73ed28716b9f8c1c4161deb1234240"
server_key_iv: "88b8ce1dd692cff21cf3a7ab60fece6e"
server_key_version: "encryptionkey@0"
I20260621 06:56:44.875751 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "b71d080af9c3449eb2760e39b3e52150"
format_stamp: "Formatted at 2026-06-21 06:56:44 on dist-test-slave-gmf2"
server_key: "0b73ed28716b9f8c1c4161deb1234240"
server_key_iv: "88b8ce1dd692cff21cf3a7ab60fece6e"
server_key_version: "encryptionkey@0"
I20260621 06:56:44.881060 7992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 06:56:44.884522 8007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:44.885790 7992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 06:56:44.886008 7992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "b71d080af9c3449eb2760e39b3e52150"
format_stamp: "Formatted at 2026-06-21 06:56:44 on dist-test-slave-gmf2"
server_key: "0b73ed28716b9f8c1c4161deb1234240"
server_key_iv: "88b8ce1dd692cff21cf3a7ab60fece6e"
server_key_version: "encryptionkey@0"
I20260621 06:56:44.886209 7992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:44.898386 7992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:44.903170 7992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:44.903473 7992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:44.904417 7992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:44.905733 7992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:44.905836 7992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:44.905997 7992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:44.906071 7992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:44.939285 7992 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:44219
I20260621 06:56:44.939338 8119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:44219 every 8 connection(s)
I20260621 06:56:44.940600 7992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:56:44.941531 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 7992
I20260621 06:56:44.941725 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:56:44.942363 20963 external_mini_cluster.cc:1428] Setting key 2159c7025b41b5a6366b4bf49b09686a
Jun 21 06:56:44 dist-test-slave-gmf2 krb5kdc[7752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025004, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.194@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:56:44.965222 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:44.943514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:60181 (local address 127.20.120.254:46361)
0621 06:56:44.943683 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:44.943687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:44.945204 (+ 1517us) server_negotiation.cc:407] Connection header received
0621 06:56:44.948112 (+ 2908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:44.948116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:44.948183 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:44.948284 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:44.950364 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:44.951110 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:44.953584 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:44.954065 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:44.954242 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:44.957566 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:44.957575 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:44.957582 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:44.957627 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:44.961867 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:44.962891 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:44.962897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:44.962901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:44.963054 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:44.963555 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:44.963558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:44.963560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:44.963824 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:44.963937 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:44.964815 (+ 878us) server_negotiation.cc:299] Negotiation successful
0621 06:56:44.964987 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 06:56:44.966846 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:44.944113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:46361 (local address 127.20.120.194:60181)
0621 06:56:44.945013 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:44.945073 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:44.947816 (+ 2743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:44.948556 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:44.948569 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:44.949340 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:44.950172 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:44.950199 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:44.951289 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:44.951297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:44.953430 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:44.953448 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:44.954300 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:44.954305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:44.954497 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:44.955191 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:56:44.955213 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:44.957340 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:44.962059 (+ 4719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:44.962070 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:44.962089 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:44.962756 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:44.963173 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:44.963181 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:44.963195 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:44.963445 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:44.963953 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:44.963964 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:44.964418 (+ 454us) client_negotiation.cc:769] Sending connection context
0621 06:56:44.964741 (+ 323us) client_negotiation.cc:240] Negotiation successful
0621 06:56:44.965330 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":340,"threads_started":1}
I20260621 06:56:44.968458 8120 heartbeater.cc:347] Connected to a master server at 127.20.120.254:46361
I20260621 06:56:44.968757 8120 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:44.969470 8120 heartbeater.cc:511] Master 127.20.120.254:46361 requested a full tablet report, sending...
I20260621 06:56:44.970973 7797 ts_manager.cc:194] Registered new tserver with Master: b71d080af9c3449eb2760e39b3e52150 (127.20.120.194:44219)
WARNING: no policy specified for kudu/127.20.120.195@KRBTEST.COM; defaulting to no policy
I20260621 06:56:44.971819 7797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.194@KRBTEST.COM'} at 127.20.120.194:60181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
Entry for principal kudu/127.20.120.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab.
I20260621 06:56:45.003324 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:46361
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:45.322839 8125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:45.323165 8125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:45.323338 8125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:45.330716 8125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:45.330866 8125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:45.331039 8125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:56:45.340849 8125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.195.keytab
--principal=kudu/127.20.120.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:46361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:45.342298 8125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:45.343681 8125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:45.357112 8130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:45.357177 8131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:45.357211 8133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:45.358040 8125 server_base.cc:1034] running on GCE node
I20260621 06:56:45.358733 8125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:45.359737 8125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:45.360930 8125 hybrid_clock.cc:648] HybridClock initialized: now 1782025005360884 us; error 39 us; skew 500 ppm
I20260621 06:56:45.361253 8125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:45 dist-test-slave-gmf2 krb5kdc[7752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025005, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:45.364328 8125 init.cc:377] Logged in from keytab as kudu/127.20.120.195@KRBTEST.COM (short username kudu)
I20260621 06:56:45.365803 8125 webserver.cc:460] Webserver started at http://127.20.120.195:34049/ using document root <none> and password file <none>
I20260621 06:56:45.366494 8125 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:45.366623 8125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:45.366956 8125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:45.369779 8125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "d3d06f5a721a45609777eee5039ff2b9"
format_stamp: "Formatted at 2026-06-21 06:56:45 on dist-test-slave-gmf2"
server_key: "35bff6101af073ba95f3bd8e1d4ed7ed"
server_key_iv: "13eb5ea1382e49c8b98a4c9084ea0a0e"
server_key_version: "encryptionkey@0"
I20260621 06:56:45.370584 8125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "d3d06f5a721a45609777eee5039ff2b9"
format_stamp: "Formatted at 2026-06-21 06:56:45 on dist-test-slave-gmf2"
server_key: "35bff6101af073ba95f3bd8e1d4ed7ed"
server_key_iv: "13eb5ea1382e49c8b98a4c9084ea0a0e"
server_key_version: "encryptionkey@0"
I20260621 06:56:45.375622 8125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260621 06:56:45.379318 8140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:45.380640 8125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 06:56:45.380800 8125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "d3d06f5a721a45609777eee5039ff2b9"
format_stamp: "Formatted at 2026-06-21 06:56:45 on dist-test-slave-gmf2"
server_key: "35bff6101af073ba95f3bd8e1d4ed7ed"
server_key_iv: "13eb5ea1382e49c8b98a4c9084ea0a0e"
server_key_version: "encryptionkey@0"
I20260621 06:56:45.381006 8125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:45.395551 8125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:45.399513 8125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:45.399799 8125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:45.400710 8125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:45.402051 8125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:45.402120 8125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:45.402251 8125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:45.402314 8125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:45.438338 8125 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:41005
I20260621 06:56:45.438378 8252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:41005 every 8 connection(s)
I20260621 06:56:45.440204 8125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:56:45.446171 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 8125
I20260621 06:56:45.446365 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:56:45.446931 20963 external_mini_cluster.cc:1428] Setting key 1f95dc3a30da5990bfd997a43764fdc7
Jun 21 06:56:45 dist-test-slave-gmf2 krb5kdc[7752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025005, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.195@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:56:45.461807 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.443759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:52961 (local address 127.20.120.254:46361)
0621 06:56:45.443963 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:45.443969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:45.445180 (+ 1211us) server_negotiation.cc:407] Connection header received
0621 06:56:45.447582 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:45.447589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:45.447671 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:45.447752 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:45.449511 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.450315 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.451828 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.452231 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.452321 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.455552 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:45.455557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:45.455561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:45.455595 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:45.458351 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:45.459267 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:45.459271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:45.459273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:45.459405 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:45.459882 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:45.459886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:45.459890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:45.460237 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:45.460403 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:45.461227 (+ 824us) server_negotiation.cc:299] Negotiation successful
0621 06:56:45.461471 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:56:45.462936 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.444312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:46361 (local address 127.20.120.195:52961)
0621 06:56:45.445016 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:45.445059 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:45.447292 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:45.448020 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:45.448031 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:45.448744 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:45.449346 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:45.449372 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.450449 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.450452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:45.451704 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:45.451715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.452414 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.452419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:45.452602 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.453205 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:56:45.453225 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:45.455369 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:45.458517 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:45.458525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:45.458541 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:45.459160 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:45.459514 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:56:45.459519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:56:45.459522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:56:45.459749 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:56:45.460401 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:45.460411 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:45.460815 (+ 404us) client_negotiation.cc:769] Sending connection context
0621 06:56:45.461093 (+ 278us) client_negotiation.cc:240] Negotiation successful
0621 06:56:45.461622 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":270,"threads_started":1}
I20260621 06:56:45.464176 8253 heartbeater.cc:347] Connected to a master server at 127.20.120.254:46361
I20260621 06:56:45.464417 8253 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:45.465024 8253 heartbeater.cc:511] Master 127.20.120.254:46361 requested a full tablet report, sending...
I20260621 06:56:45.466270 7797 ts_manager.cc:194] Registered new tserver with Master: d3d06f5a721a45609777eee5039ff2b9 (127.20.120.195:41005)
I20260621 06:56:45.466965 7797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.120.195@KRBTEST.COM'} at 127.20.120.195:52961
I20260621 06:56:45.474222 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:56:45.490222 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.478253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.20.120.254:46361)
0621 06:56:45.478402 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:45.478406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:45.478832 (+ 426us) server_negotiation.cc:407] Connection header received
0621 06:56:45.479006 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:45.479009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:45.479066 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:45.479143 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:45.480457 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.481155 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.482540 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.482940 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.483027 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.485360 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:45.485365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:45.485367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:45.485413 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:45.487707 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:45.488277 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:45.488280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:45.488282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:45.488417 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:45.488754 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:45.488759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:45.488761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:45.489165 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:45.489409 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:45.489750 (+ 341us) server_negotiation.cc:299] Negotiation successful
0621 06:56:45.489941 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 06:56:45.490407 7986 heartbeater.cc:503] Master 127.20.120.254:46361 was elected leader, sending a full tablet report...
I20260621 06:56:45.494019 7796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:56:45.502792 7796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:56:45.534216 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.516141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:41005 (local address 127.0.0.1:49862)
0621 06:56:45.523506 (+ 7365us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:45.523525 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:45.523642 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:45.525097 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:45.525108 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:45.525153 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:45.525642 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:45.525654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.528264 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.528272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:45.531188 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:45.531200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.533205 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.533210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:45.533366 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.533470 (+ 104us) client_negotiation.cc:769] Sending connection context
0621 06:56:45.533639 (+ 169us) client_negotiation.cc:240] Negotiation successful
0621 06:56:45.533819 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7243,"thread_start_us":343,"threads_started":1}
I20260621 06:56:45.534216 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.516319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.20.120.195:41005)
0621 06:56:45.524307 (+ 7988us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:45.524319 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:45.524362 (+ 43us) server_negotiation.cc:407] Connection header received
0621 06:56:45.524489 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:45.524500 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:45.524812 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:45.525062 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:45.525860 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.528006 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.531426 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.532867 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.533041 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.533232 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:45.533718 (+ 486us) server_negotiation.cc:299] Negotiation successful
0621 06:56:45.533881 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7742,"thread_start_us":7590,"threads_started":1}
I20260621 06:56:45.536036 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.516020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.20.120.194:44219)
0621 06:56:45.525805 (+ 9785us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:45.525816 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:45.525851 (+ 35us) server_negotiation.cc:407] Connection header received
0621 06:56:45.526003 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:45.526015 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:45.527504 (+ 1489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:45.528628 (+ 1124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:45.528897 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.530795 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.533140 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.534907 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.535076 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.535228 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:45.535535 (+ 307us) server_negotiation.cc:299] Negotiation successful
0621 06:56:45.535689 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9577,"thread_start_us":9457,"threads_started":1}
I20260621 06:56:45.536042 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.515800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:44219 (local address 127.0.0.1:34854)
0621 06:56:45.516816 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:45.516850 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:45.517029 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:45.527763 (+ 10734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:45.527771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:45.527817 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:45.528373 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:45.528385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.531002 (+ 2617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.531007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:45.532974 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:45.532987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.535118 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.535124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:45.535308 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.535383 (+ 75us) client_negotiation.cc:769] Sending connection context
0621 06:56:45.535473 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 06:56:45.535598 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":5,"thread_start_us":7460,"threads_started":1}
I20260621 06:56:45.538476 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec3be2278554d82a08fa5193abd2531 (DEFAULT_TABLE table=test-table [id=030363888a0d40a08be04d6600d35f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:45.539422 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.525805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:34631 (local address 127.0.0.1:50742)
0621 06:56:45.531165 (+ 5360us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:45.531184 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:45.531328 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:45.531990 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:45.531995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:45.532020 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:45.532408 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:45.532418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.534409 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.534414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:45.536685 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:45.536696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.538647 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.538653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:45.538798 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.538827 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 06:56:45.538925 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 06:56:45.539035 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5199,"mutex_wait_us":120,"thread_start_us":5085,"threads_started":1}
I20260621 06:56:45.539979 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec3be2278554d82a08fa5193abd2531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:45.540138 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.523929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.20.120.193:34631)
0621 06:56:45.530871 (+ 6942us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:45.530879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:45.531455 (+ 576us) server_negotiation.cc:407] Connection header received
0621 06:56:45.531560 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:45.531568 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:45.531787 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:45.531975 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:45.532607 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.534222 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.536831 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.538169 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.538348 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.538498 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:45.539677 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0621 06:56:45.539843 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6763,"thread_start_us":6600,"threads_started":1}
I20260621 06:56:45.539776 8055 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec3be2278554d82a08fa5193abd2531 (DEFAULT_TABLE table=test-table [id=030363888a0d40a08be04d6600d35f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:45.541757 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec3be2278554d82a08fa5193abd2531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:45.543383 7921 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec3be2278554d82a08fa5193abd2531 (DEFAULT_TABLE table=test-table [id=030363888a0d40a08be04d6600d35f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:45.544699 7921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec3be2278554d82a08fa5193abd2531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:45.550851 8271 tablet_bootstrap.cc:492] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9: Bootstrap starting.
I20260621 06:56:45.554184 8271 tablet_bootstrap.cc:654] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:45.555446 8271 log.cc:824] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:45.556434 8272 tablet_bootstrap.cc:492] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150: Bootstrap starting.
I20260621 06:56:45.558813 8274 tablet_bootstrap.cc:492] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292: Bootstrap starting.
I20260621 06:56:45.559773 8271 tablet_bootstrap.cc:492] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9: No bootstrap required, opened a new log
I20260621 06:56:45.560038 8271 ts_tablet_manager.cc:1397] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 06:56:45.560105 8272 tablet_bootstrap.cc:654] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:45.561292 8272 log.cc:824] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:45.561453 8274 tablet_bootstrap.cc:654] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:45.562674 8274 log.cc:824] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:45.564301 8272 tablet_bootstrap.cc:492] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150: No bootstrap required, opened a new log
I20260621 06:56:45.564564 8272 ts_tablet_manager.cc:1397] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260621 06:56:45.565233 8274 tablet_bootstrap.cc:492] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292: No bootstrap required, opened a new log
I20260621 06:56:45.565467 8274 ts_tablet_manager.cc:1397] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260621 06:56:45.570539 8271 raft_consensus.cc:348] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71d080af9c3449eb2760e39b3e52150" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44219 } } peers { permanent_uuid: "d3d06f5a721a45609777eee5039ff2b9" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41005 } } peers { permanent_uuid: "e306100a914342babf22933a23b86292" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 34631 } }
I20260621 06:56:45.570968 8271 raft_consensus.cc:374] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:45.571061 8271 raft_consensus.cc:729] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d06f5a721a45609777eee5039ff2b9, State: Initialized, Role: FOLLOWER
I20260621 06:56:45.571699 8271 consensus_queue.cc:260] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71d080af9c3449eb2760e39b3e52150" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44219 } } peers { permanent_uuid: "d3d06f5a721a45609777eee5039ff2b9" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41005 } } peers { permanent_uuid: "e306100a914342babf22933a23b86292" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 34631 } }
I20260621 06:56:45.572801 8253 heartbeater.cc:503] Master 127.20.120.254:46361 was elected leader, sending a full tablet report...
I20260621 06:56:45.572685 8274 raft_consensus.cc:348] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71d080af9c3449eb2760e39b3e52150" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44219 } } peers { permanent_uuid: "d3d06f5a721a45609777eee5039ff2b9" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41005 } } peers { permanent_uuid: "e306100a914342babf22933a23b86292" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 34631 } }
I20260621 06:56:45.573050 8274 raft_consensus.cc:374] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:45.573164 8274 raft_consensus.cc:729] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e306100a914342babf22933a23b86292, State: Initialized, Role: FOLLOWER
I20260621 06:56:45.573717 8274 consensus_queue.cc:260] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71d080af9c3449eb2760e39b3e52150" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44219 } } peers { permanent_uuid: "d3d06f5a721a45609777eee5039ff2b9" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41005 } } peers { permanent_uuid: "e306100a914342babf22933a23b86292" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 34631 } }
I20260621 06:56:45.574934 8271 ts_tablet_manager.cc:1428] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260621 06:56:45.574997 8272 raft_consensus.cc:348] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71d080af9c3449eb2760e39b3e52150" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44219 } } peers { permanent_uuid: "d3d06f5a721a45609777eee5039ff2b9" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41005 } } peers { permanent_uuid: "e306100a914342babf22933a23b86292" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 34631 } }
I20260621 06:56:45.575403 8272 raft_consensus.cc:374] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:45.575510 8272 raft_consensus.cc:729] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71d080af9c3449eb2760e39b3e52150, State: Initialized, Role: FOLLOWER
I20260621 06:56:45.576220 8272 consensus_queue.cc:260] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71d080af9c3449eb2760e39b3e52150" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44219 } } peers { permanent_uuid: "d3d06f5a721a45609777eee5039ff2b9" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41005 } } peers { permanent_uuid: "e306100a914342babf22933a23b86292" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 34631 } }
I20260621 06:56:45.576604 8274 ts_tablet_manager.cc:1428] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260621 06:56:45.578521 8120 heartbeater.cc:503] Master 127.20.120.254:46361 was elected leader, sending a full tablet report...
I20260621 06:56:45.579047 8272 ts_tablet_manager.cc:1428] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
W20260621 06:56:45.692816 8121 tablet.cc:2369] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:56:45.692981 8254 tablet.cc:2369] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:56:45.711058 7987 tablet.cc:2369] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:56:45.732964 8279 raft_consensus.cc:482] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:56:45.733325 8279 raft_consensus.cc:504] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71d080af9c3449eb2760e39b3e52150" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44219 } } peers { permanent_uuid: "d3d06f5a721a45609777eee5039ff2b9" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41005 } } peers { permanent_uuid: "e306100a914342babf22933a23b86292" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 34631 } }
I20260621 06:56:45.735307 8279 leader_election.cc:288] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3d06f5a721a45609777eee5039ff2b9 (127.20.120.195:41005), e306100a914342babf22933a23b86292 (127.20.120.193:34631)
I20260621 06:56:45.744715 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.735799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:49809 (local address 127.20.120.195:41005)
0621 06:56:45.736019 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:45.736025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:45.736810 (+ 785us) server_negotiation.cc:407] Connection header received
0621 06:56:45.737006 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:45.737012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:45.737091 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:45.737192 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:45.738063 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.739177 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.742303 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.743542 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.743712 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.743788 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:45.744326 (+ 538us) server_negotiation.cc:299] Negotiation successful
0621 06:56:45.744455 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260621 06:56:45.744716 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.735921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:50607 (local address 127.20.120.193:34631)
0621 06:56:45.736126 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:45.736133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:45.736810 (+ 677us) server_negotiation.cc:407] Connection header received
0621 06:56:45.737006 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:45.737012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:45.737091 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:45.737191 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:45.738097 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.739165 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.742302 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.743549 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.743712 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.743788 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:45.744327 (+ 539us) server_negotiation.cc:299] Negotiation successful
0621 06:56:45.744484 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 06:56:45.744935 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.735610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:41005 (local address 127.20.120.194:49809)
0621 06:56:45.736649 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:45.736671 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:45.736851 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:45.737224 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:45.737229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:45.737260 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:45.737867 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:45.737882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.739307 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.739312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:45.742130 (+ 2818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:45.742147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.743839 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.743846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:45.744109 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.744188 (+ 79us) client_negotiation.cc:769] Sending connection context
0621 06:56:45.744287 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 06:56:45.744451 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":292,"spinlock_wait_cycles":64128,"thread_start_us":677,"threads_started":1}
I20260621 06:56:45.745307 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.735709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:34631 (local address 127.20.120.194:50607)
0621 06:56:45.736687 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:45.736704 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:45.736850 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:45.737224 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:45.737229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:45.737267 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:45.737939 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:45.737952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.739307 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.739312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:45.742130 (+ 2818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:45.742147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.743839 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.743846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:45.744110 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.744194 (+ 84us) client_negotiation.cc:769] Sending connection context
0621 06:56:45.744287 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 06:56:45.744451 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":354,"spinlock_wait_cycles":3840,"thread_start_us":485,"threads_started":1}
I20260621 06:56:45.746191 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec3be2278554d82a08fa5193abd2531" candidate_uuid: "b71d080af9c3449eb2760e39b3e52150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d06f5a721a45609777eee5039ff2b9" is_pre_election: true
I20260621 06:56:45.746429 7941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec3be2278554d82a08fa5193abd2531" candidate_uuid: "b71d080af9c3449eb2760e39b3e52150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e306100a914342babf22933a23b86292" is_pre_election: true
I20260621 06:56:45.746800 8208 raft_consensus.cc:2454] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b71d080af9c3449eb2760e39b3e52150 in term 0.
I20260621 06:56:45.746955 7941 raft_consensus.cc:2454] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b71d080af9c3449eb2760e39b3e52150 in term 0.
I20260621 06:56:45.747622 8009 leader_election.cc:302] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b71d080af9c3449eb2760e39b3e52150, d3d06f5a721a45609777eee5039ff2b9, e306100a914342babf22933a23b86292; no voters:
I20260621 06:56:45.748095 8279 raft_consensus.cc:2781] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:56:45.748250 8279 raft_consensus.cc:482] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:56:45.748382 8279 raft_consensus.cc:3037] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:45.750732 8279 raft_consensus.cc:504] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71d080af9c3449eb2760e39b3e52150" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44219 } } peers { permanent_uuid: "d3d06f5a721a45609777eee5039ff2b9" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41005 } } peers { permanent_uuid: "e306100a914342babf22933a23b86292" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 34631 } }
I20260621 06:56:45.751459 8279 leader_election.cc:288] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [CANDIDATE]: Term 1 election: Requested vote from peers d3d06f5a721a45609777eee5039ff2b9 (127.20.120.195:41005), e306100a914342babf22933a23b86292 (127.20.120.193:34631)
I20260621 06:56:45.751959 7941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec3be2278554d82a08fa5193abd2531" candidate_uuid: "b71d080af9c3449eb2760e39b3e52150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e306100a914342babf22933a23b86292"
I20260621 06:56:45.751959 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec3be2278554d82a08fa5193abd2531" candidate_uuid: "b71d080af9c3449eb2760e39b3e52150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d06f5a721a45609777eee5039ff2b9"
I20260621 06:56:45.752177 7941 raft_consensus.cc:3037] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:45.752177 8208 raft_consensus.cc:3037] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:45.755394 8208 raft_consensus.cc:2454] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b71d080af9c3449eb2760e39b3e52150 in term 1.
I20260621 06:56:45.755394 7941 raft_consensus.cc:2454] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b71d080af9c3449eb2760e39b3e52150 in term 1.
I20260621 06:56:45.755825 8009 leader_election.cc:302] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b71d080af9c3449eb2760e39b3e52150, d3d06f5a721a45609777eee5039ff2b9, e306100a914342babf22933a23b86292; no voters:
I20260621 06:56:45.756101 8279 raft_consensus.cc:2781] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:45.756533 8279 raft_consensus.cc:686] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 1 LEADER]: Becoming Leader. State: Replica: b71d080af9c3449eb2760e39b3e52150, State: Running, Role: LEADER
I20260621 06:56:45.757047 8279 consensus_queue.cc:237] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71d080af9c3449eb2760e39b3e52150" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44219 } } peers { permanent_uuid: "d3d06f5a721a45609777eee5039ff2b9" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41005 } } peers { permanent_uuid: "e306100a914342babf22933a23b86292" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 34631 } }
I20260621 06:56:45.761494 7797 catalog_manager.cc:5495] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 reported cstate change: term changed from 0 to 1, leader changed from <none> to b71d080af9c3449eb2760e39b3e52150 (127.20.120.194). New cstate: current_term: 1 leader_uuid: "b71d080af9c3449eb2760e39b3e52150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71d080af9c3449eb2760e39b3e52150" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 44219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3d06f5a721a45609777eee5039ff2b9" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e306100a914342babf22933a23b86292" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 34631 } health_report { overall_health: UNKNOWN } } }
I20260621 06:56:45.821708 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.816127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34858 (local address 127.20.120.194:44219)
0621 06:56:45.816350 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:45.816356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:45.816468 (+ 112us) server_negotiation.cc:407] Connection header received
0621 06:56:45.816717 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:45.816721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:45.816807 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:45.816998 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:56:45.817366 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.818183 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.819539 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.819961 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.820090 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.820576 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:56:45.821074 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:56:45.821231 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:45.821333 (+ 102us) server_negotiation.cc:299] Negotiation successful
0621 06:56:45.821424 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 06:56:45.829772 7941 raft_consensus.cc:1261] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292 [term 1 FOLLOWER]: Refusing update from remote peer b71d080af9c3449eb2760e39b3e52150: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:56:45.829771 8208 raft_consensus.cc:1261] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9 [term 1 FOLLOWER]: Refusing update from remote peer b71d080af9c3449eb2760e39b3e52150: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:56:45.830986 8279 consensus_queue.cc:1035] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e306100a914342babf22933a23b86292" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 34631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:56:45.831327 8282 consensus_queue.cc:1035] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3d06f5a721a45609777eee5039ff2b9" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:56:45.843540 8285 mvcc.cc:205] Tried to move back new op lower bound from 7299174423868715008 to 7299174423585046528. Current Snapshot: MvccSnapshot[applied={T|T < 7299174423868715008}]
I20260621 06:56:45.844343 8287 mvcc.cc:205] Tried to move back new op lower bound from 7299174423868715008 to 7299174423585046528. Current Snapshot: MvccSnapshot[applied={T|T < 7299174423868715008}]
I20260621 06:56:45.860757 7796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:56:45.861123 7796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:56:45.865051 7796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9951e3f8ee34e6aaf16b0342024d825: Sending DeleteTablet for 3 replicas of tablet 4ec3be2278554d82a08fa5193abd2531
I20260621 06:56:45.866153 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ec3be2278554d82a08fa5193abd2531 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:45 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:34854
I20260621 06:56:45.866358 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ec3be2278554d82a08fa5193abd2531 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:45 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:49862
I20260621 06:56:45.867913 8296 tablet_replica.cc:330] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9: stopping tablet replica
I20260621 06:56:45.868407 8296 raft_consensus.cc:2229] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:56:45.868703 7921 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ec3be2278554d82a08fa5193abd2531 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:45 UTC) from {username='kudu', principal='kudu/127.20.120.254@KRBTEST.COM'} at 127.0.0.1:50742
I20260621 06:56:45.869057 8296 raft_consensus.cc:2258] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:45.871362 8296 ts_tablet_manager.cc:1905] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:45.872475 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 7858
I20260621 06:56:45.879395 8296 ts_tablet_manager.cc:1918] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:45.879683 8296 log.cc:1196] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/4ec3be2278554d82a08fa5193abd2531
I20260621 06:56:45.880281 8296 ts_tablet_manager.cc:1939] T 4ec3be2278554d82a08fa5193abd2531 P d3d06f5a721a45609777eee5039ff2b9: Deleting consensus metadata
I20260621 06:56:45.882402 7781 catalog_manager.cc:4841] TS d3d06f5a721a45609777eee5039ff2b9 (127.20.120.195:41005): tablet 4ec3be2278554d82a08fa5193abd2531 (table test-table [id=030363888a0d40a08be04d6600d35f25]) successfully deleted
I20260621 06:56:45.885432 8297 tablet_replica.cc:330] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292: stopping tablet replica
I20260621 06:56:45.885942 8297 raft_consensus.cc:2229] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:56:45.886644 8297 raft_consensus.cc:2258] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:45.887593 8295 tablet_replica.cc:330] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150: stopping tablet replica
I20260621 06:56:45.888068 8295 raft_consensus.cc:2229] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:56:45.888883 8295 raft_consensus.cc:2258] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:45.891175 8295 ts_tablet_manager.cc:1905] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 06:56:45 dist-test-slave-gmf2 krb5kdc[7752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.193@KRBTEST.COM
I20260621 06:56:45.899291 8295 ts_tablet_manager.cc:1918] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:45.899497 8295 log.cc:1196] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/4ec3be2278554d82a08fa5193abd2531
I20260621 06:56:45.900138 8295 ts_tablet_manager.cc:1939] T 4ec3be2278554d82a08fa5193abd2531 P b71d080af9c3449eb2760e39b3e52150: Deleting consensus metadata
I20260621 06:56:45.900686 8297 ts_tablet_manager.cc:1905] T 4ec3be2278554d82a08fa5193abd2531 P e306100a914342babf22933a23b86292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:45.902096 7782 catalog_manager.cc:4841] TS b71d080af9c3449eb2760e39b3e52150 (127.20.120.194:44219): tablet 4ec3be2278554d82a08fa5193abd2531 (table test-table [id=030363888a0d40a08be04d6600d35f25]) successfully deleted
I20260621 06:56:45.902490 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:45.874423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50754 (local address 127.20.120.193:34631)
0621 06:56:45.874598 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:45.874607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:45.875353 (+ 746us) server_negotiation.cc:407] Connection header received
0621 06:56:45.876309 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:45.876319 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:45.876463 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:45.876560 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:45.881077 (+ 4517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.882537 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.886791 (+ 4254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:45.887511 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:45.887629 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:45.894071 (+ 6442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:45.894092 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:45.894112 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:45.894172 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:45.898164 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:45.899097 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:45.899109 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:45.899112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:45.899289 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:45.899833 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:45.899838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:45.899841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:45.900342 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:45.900677 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:45.901266 (+ 589us) server_negotiation.cc:299] Negotiation successful
0621 06:56:45.901536 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 06:56:45.903128 7981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:45.993857 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 7858
W20260621 06:56:46.012157 7782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.120.193:34631 (error 108)
I20260621 06:56:46.012408 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 7992
I20260621 06:56:46.015326 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:46.014780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:34631 (local address 127.0.0.1:50762)
0621 06:56:46.014981 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:46.015132 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260621 06:56:46.015643 7782 catalog_manager.cc:4584] TS e306100a914342babf22933a23b86292 (127.20.120.193:34631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ec3be2278554d82a08fa5193abd2531: Network error: Client connection negotiation failed: client connection to 127.20.120.193:34631: connect: Connection refused (error 111)
Jun 21 06:56:46 dist-test-slave-gmf2 krb5kdc[7752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.194@KRBTEST.COM
I20260621 06:56:46.031143 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:46.013397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.20.120.194:44219)
0621 06:56:46.013581 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:46.013592 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:46.013886 (+ 294us) server_negotiation.cc:407] Connection header received
0621 06:56:46.013998 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:46.014003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:46.014098 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:46.014223 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:46.015953 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:46.017223 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:46.018563 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:46.019284 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:46.019439 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:46.024204 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:46.024227 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:46.024247 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:46.024301 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:46.028139 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:46.028836 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:46.028846 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:46.028850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:46.029022 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:46.029410 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:46.029416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:46.029419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:46.029868 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:46.030201 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:46.030494 (+ 293us) server_negotiation.cc:299] Negotiation successful
0621 06:56:46.030720 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 06:56:46.031869 8115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:46.069350 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:46.068908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:34631 (local address 127.0.0.1:50766)
0621 06:56:46.069124 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:46.069212 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260621 06:56:46.070390 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:46.070060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:34631 (local address 127.0.0.1:50768)
0621 06:56:46.070202 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:46.070285 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 06:56:46.130774 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:46.130293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:34631 (local address 127.0.0.1:50770)
0621 06:56:46.130523 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:46.130606 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 06:56:46.131745 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:46.131418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:34631 (local address 127.0.0.1:50774)
0621 06:56:46.131578 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:46.131642 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 06:56:46.142413 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 7992
I20260621 06:56:46.162030 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 8125
Jun 21 06:56:46 dist-test-slave-gmf2 krb5kdc[7752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.195@KRBTEST.COM
I20260621 06:56:46.176388 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:46.162808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.20.120.195:41005)
0621 06:56:46.162987 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:46.162991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:46.163008 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:56:46.163155 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:46.163159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:46.163220 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:46.163301 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:46.164491 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:46.165180 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:46.166549 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:46.166958 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:46.167052 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:46.171265 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:46.171297 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:46.171321 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:46.171356 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:46.173854 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:46.174570 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:46.174578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:46.174580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:46.174722 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:46.175039 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:46.175042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:46.175044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:46.175371 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:46.175581 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:46.175855 (+ 274us) server_negotiation.cc:299] Negotiation successful
0621 06:56:46.176021 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 06:56:46.177021 8248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:46.212639 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:46.212086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:34631 (local address 127.0.0.1:50788)
0621 06:56:46.212334 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:46.212461 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260621 06:56:46.213724 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:46.213323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:34631 (local address 127.0.0.1:50790)
0621 06:56:46.213543 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:46.213617 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260621 06:56:46.266481 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 8125
I20260621 06:56:46.283113 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 7765
I20260621 06:56:46.284255 7827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:46.394430 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:46.393710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:34631 (local address 127.0.0.1:50796)
0621 06:56:46.393976 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:46.394107 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:34631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260621 06:56:46.394613 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 7765
2026-06-21T06:56:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4242 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-21T06:56:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:56:46Z Disabled control of system clock
I20260621 06:56:46.470305 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:33391
--webserver_interface=127.20.120.254
--webserver_port=0
--builtin_ntp_servers=127.20.120.212:33029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:33391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 06:56:46.796748 8307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:46.797044 8307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:46.797201 8307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:46.805230 8307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:56:46.805341 8307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:46.805438 8307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:46.805527 8307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:56:46.805614 8307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:56:46.815114 8307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:33029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:33391
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:33391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:46.816427 8307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:46.818397 8307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:46.830432 8312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:46.830498 8313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:46.830869 8315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:46.831593 8307 server_base.cc:1034] running on GCE node
I20260621 06:56:46.832418 8307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:46.833999 8307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:46.835246 8307 hybrid_clock.cc:648] HybridClock initialized: now 1782025006835187 us; error 52 us; skew 500 ppm
I20260621 06:56:46.835642 8307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:56:46.838169 8307 webserver.cc:460] Webserver started at http://127.20.120.254:43741/ using document root <none> and password file <none>
I20260621 06:56:46.838928 8307 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:46.839068 8307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:46.839403 8307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:46.842681 8307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "638524c1674541a0a821083c14177d98"
format_stamp: "Formatted at 2026-06-21 06:56:46 on dist-test-slave-gmf2"
server_key: "dcad6c4f3c286ba310d89baba79f7d83"
server_key_iv: "2fc174f96150c9040b69b3f5564cf2c6"
server_key_version: "encryptionkey@0"
I20260621 06:56:46.843565 8307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "638524c1674541a0a821083c14177d98"
format_stamp: "Formatted at 2026-06-21 06:56:46 on dist-test-slave-gmf2"
server_key: "dcad6c4f3c286ba310d89baba79f7d83"
server_key_iv: "2fc174f96150c9040b69b3f5564cf2c6"
server_key_version: "encryptionkey@0"
I20260621 06:56:46.849483 8307 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260621 06:56:46.853715 8321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:46.855180 8307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:56:46.855372 8307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "638524c1674541a0a821083c14177d98"
format_stamp: "Formatted at 2026-06-21 06:56:46 on dist-test-slave-gmf2"
server_key: "dcad6c4f3c286ba310d89baba79f7d83"
server_key_iv: "2fc174f96150c9040b69b3f5564cf2c6"
server_key_version: "encryptionkey@0"
I20260621 06:56:46.855594 8307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:46.877454 8307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:46.878350 8307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:46.878670 8307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:46.896831 8307 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:33391
I20260621 06:56:46.896874 8372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:33391 every 8 connection(s)
I20260621 06:56:46.898201 8307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:56:46.901930 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 8307
I20260621 06:56:46.902000 8373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:46.902120 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:56:46.902571 20963 external_mini_cluster.cc:1428] Setting key f6874665160241893af2b1818db557a9
I20260621 06:56:46.915040 8373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98: Bootstrap starting.
I20260621 06:56:46.916004 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:46.904443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.20.120.254:33391)
0621 06:56:46.905499 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:46.905513 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:46.905556 (+ 43us) server_negotiation.cc:407] Connection header received
0621 06:56:46.907206 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:46.907227 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:46.907701 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:46.908126 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:46.908733 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:46.910071 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:46.911463 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:46.912179 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:46.912402 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:46.913634 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:46.913641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:46.913655 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:46.914266 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:46.914358 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:46.914379 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:46.914517 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:46.914741 (+ 224us) server_negotiation.cc:299] Negotiation successful
0621 06:56:46.915046 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":399,"threads_started":1}
I20260621 06:56:46.918279 8373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:46.919414 8373 log.cc:824] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:46.922313 8373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98: No bootstrap required, opened a new log
I20260621 06:56:46.929836 8373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638524c1674541a0a821083c14177d98" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 33391 } }
I20260621 06:56:46.930363 8373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:46.930446 8373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638524c1674541a0a821083c14177d98, State: Initialized, Role: FOLLOWER
I20260621 06:56:46.930924 8373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638524c1674541a0a821083c14177d98" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 33391 } }
I20260621 06:56:46.931120 8373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:56:46.931224 8373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:56:46.931337 8373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:46.933326 8373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638524c1674541a0a821083c14177d98" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 33391 } }
I20260621 06:56:46.933758 8373 leader_election.cc:302] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 638524c1674541a0a821083c14177d98; no voters:
I20260621 06:56:46.934335 8373 leader_election.cc:288] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:56:46.934497 8378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:46.935271 8378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [term 1 LEADER]: Becoming Leader. State: Replica: 638524c1674541a0a821083c14177d98, State: Running, Role: LEADER
I20260621 06:56:46.935798 8378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638524c1674541a0a821083c14177d98" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 33391 } }
I20260621 06:56:46.936360 8373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:56:46.938777 8380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 638524c1674541a0a821083c14177d98. Latest consensus state: current_term: 1 leader_uuid: "638524c1674541a0a821083c14177d98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638524c1674541a0a821083c14177d98" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 33391 } } }
I20260621 06:56:46.939118 8380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:46.939518 8379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "638524c1674541a0a821083c14177d98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638524c1674541a0a821083c14177d98" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 33391 } } }
I20260621 06:56:46.939946 8379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:46.945623 8384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:56:46.951326 8384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:56:46.959440 8384 catalog_manager.cc:1347] Generated new cluster ID: cb6978d5c2c74c21935d4d6e34a941c0
I20260621 06:56:46.959546 8384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:56:46.975431 8384 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:56:46.977321 8384 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:56:46.991878 8384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98: Generated new TSK 0
I20260621 06:56:46.992974 8384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:56:47.010766 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.193:0
--local_ip_for_outbound_sockets=127.20.120.193
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:33391
--builtin_ntp_servers=127.20.120.212:33029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 06:56:47.338801 8397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:47.339089 8397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:47.339183 8397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:47.346520 8397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:47.346642 8397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:47.346756 8397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.193
I20260621 06:56:47.356071 8397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:33029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.193
--webserver_port=0
--tserver_master_addrs=127.20.120.254:33391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.193
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:47.357429 8397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:47.359352 8397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:47.372337 8403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:47.372370 8405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:47.372320 8402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:47.373383 8397 server_base.cc:1034] running on GCE node
I20260621 06:56:47.374037 8397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:47.374913 8397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:47.376116 8397 hybrid_clock.cc:648] HybridClock initialized: now 1782025007376078 us; error 32 us; skew 500 ppm
I20260621 06:56:47.376456 8397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:56:47.378774 8397 webserver.cc:460] Webserver started at http://127.20.120.193:41431/ using document root <none> and password file <none>
I20260621 06:56:47.379475 8397 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:47.379592 8397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:47.379925 8397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:47.382968 8397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/data/instance:
uuid: "a7dfda97ec9d4fa698dfe5778059083a"
format_stamp: "Formatted at 2026-06-21 06:56:47 on dist-test-slave-gmf2"
server_key: "221e422fa7787df6556875185f73eb04"
server_key_iv: "d3c7c8e8441b2fc73fb33a9b90eb28a9"
server_key_version: "encryptionkey@0"
I20260621 06:56:47.383718 8397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance:
uuid: "a7dfda97ec9d4fa698dfe5778059083a"
format_stamp: "Formatted at 2026-06-21 06:56:47 on dist-test-slave-gmf2"
server_key: "221e422fa7787df6556875185f73eb04"
server_key_iv: "d3c7c8e8441b2fc73fb33a9b90eb28a9"
server_key_version: "encryptionkey@0"
I20260621 06:56:47.388525 8397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 06:56:47.391786 8411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:47.393103 8397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 06:56:47.393270 8397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/wal
uuid: "a7dfda97ec9d4fa698dfe5778059083a"
format_stamp: "Formatted at 2026-06-21 06:56:47 on dist-test-slave-gmf2"
server_key: "221e422fa7787df6556875185f73eb04"
server_key_iv: "d3c7c8e8441b2fc73fb33a9b90eb28a9"
server_key_version: "encryptionkey@0"
I20260621 06:56:47.393484 8397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:47.410086 8397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:47.410837 8397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:47.411115 8397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:47.412019 8397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:47.413328 8397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:47.413460 8397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:47.413578 8397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:47.413664 8397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:47.449499 8397 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.193:43489
I20260621 06:56:47.449579 8523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.193:43489 every 8 connection(s)
I20260621 06:56:47.450788 8397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/data/info.pb
I20260621 06:56:47.454452 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 8397
I20260621 06:56:47.454636 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-0/wal/instance
I20260621 06:56:47.455152 20963 external_mini_cluster.cc:1428] Setting key 083468058d5257dc7f425f327559c12e
I20260621 06:56:47.457798 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.194:0
--local_ip_for_outbound_sockets=127.20.120.194
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:33391
--builtin_ntp_servers=127.20.120.212:33029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 06:56:47.464208 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:47.453303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.193:37931 (local address 127.20.120.254:33391)
0621 06:56:47.453682 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:47.453686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:47.454987 (+ 1301us) server_negotiation.cc:407] Connection header received
0621 06:56:47.456884 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:47.456888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:47.456981 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:47.457090 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:47.458177 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:47.459413 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:47.461104 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:47.461833 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:47.461999 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:47.463255 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:47.463264 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:47.463271 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:47.463295 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:47.463346 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:47.463363 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:47.463455 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:47.463659 (+ 204us) server_negotiation.cc:299] Negotiation successful
0621 06:56:47.463770 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":192,"threads_started":1}
I20260621 06:56:47.464915 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:47.453859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:33391 (local address 127.20.120.193:37931)
0621 06:56:47.454845 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:47.454880 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:47.456599 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:47.457288 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:47.457296 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:47.457381 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:47.458022 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:47.458042 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:47.459554 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:47.459558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:47.460964 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:47.460976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:47.462065 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:47.462070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:47.462262 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:47.462951 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:56:47.462954 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:47.462992 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:56:47.463069 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:47.463459 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:47.463467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:47.463472 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 06:56:47.463600 (+ 128us) client_negotiation.cc:240] Negotiation successful
0621 06:56:47.463884 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":283,"threads_started":1}
I20260621 06:56:47.466598 8524 heartbeater.cc:347] Connected to a master server at 127.20.120.254:33391
I20260621 06:56:47.466913 8524 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:47.467757 8524 heartbeater.cc:511] Master 127.20.120.254:33391 requested a full tablet report, sending...
I20260621 06:56:47.469815 8338 ts_manager.cc:194] Registered new tserver with Master: a7dfda97ec9d4fa698dfe5778059083a (127.20.120.193:43489)
I20260621 06:56:47.471309 8338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.193:37931
W20260621 06:56:47.774560 8528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:47.774802 8528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:47.774940 8528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:47.782150 8528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:47.782244 8528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:47.782451 8528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.194
I20260621 06:56:47.791414 8528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:33029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.194
--webserver_port=0
--tserver_master_addrs=127.20.120.254:33391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.194
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:47.792660 8528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:47.794407 8528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:47.807098 8533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:47.807071 8534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:47.807050 8536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:47.808059 8528 server_base.cc:1034] running on GCE node
I20260621 06:56:47.808583 8528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:47.809414 8528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:47.810622 8528 hybrid_clock.cc:648] HybridClock initialized: now 1782025007810585 us; error 29 us; skew 500 ppm
I20260621 06:56:47.810971 8528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:56:47.813117 8528 webserver.cc:460] Webserver started at http://127.20.120.194:42187/ using document root <none> and password file <none>
I20260621 06:56:47.814044 8528 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:47.814173 8528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:47.814531 8528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:47.817210 8528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/data/instance:
uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5"
format_stamp: "Formatted at 2026-06-21 06:56:47 on dist-test-slave-gmf2"
server_key: "fbe332275e26487fbdaa8ea6b6ff2423"
server_key_iv: "e83cc20b805ed4f926bb891c50dfa916"
server_key_version: "encryptionkey@0"
I20260621 06:56:47.818032 8528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance:
uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5"
format_stamp: "Formatted at 2026-06-21 06:56:47 on dist-test-slave-gmf2"
server_key: "fbe332275e26487fbdaa8ea6b6ff2423"
server_key_iv: "e83cc20b805ed4f926bb891c50dfa916"
server_key_version: "encryptionkey@0"
I20260621 06:56:47.822765 8528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 06:56:47.825979 8542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:47.827179 8528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 06:56:47.827363 8528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/wal
uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5"
format_stamp: "Formatted at 2026-06-21 06:56:47 on dist-test-slave-gmf2"
server_key: "fbe332275e26487fbdaa8ea6b6ff2423"
server_key_iv: "e83cc20b805ed4f926bb891c50dfa916"
server_key_version: "encryptionkey@0"
I20260621 06:56:47.827564 8528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:47.857106 8528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:47.858029 8528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:47.858331 8528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:47.859452 8528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:47.861073 8528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:47.861196 8528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:47.861338 8528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:47.861459 8528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:47.893647 8528 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.194:36901
I20260621 06:56:47.893759 8654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.194:36901 every 8 connection(s)
I20260621 06:56:47.894948 8528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/data/info.pb
I20260621 06:56:47.904318 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 8528
I20260621 06:56:47.904547 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/wal/instance
I20260621 06:56:47.905052 20963 external_mini_cluster.cc:1428] Setting key d1c9180d740c62559780a48c9cd50e09
I20260621 06:56:47.907637 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.195:0
--local_ip_for_outbound_sockets=127.20.120.195
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:33391
--builtin_ntp_servers=127.20.120.212:33029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 06:56:47.915406 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:47.897503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.194:48721 (local address 127.20.120.254:33391)
0621 06:56:47.897732 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:47.897749 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:47.905456 (+ 7707us) server_negotiation.cc:407] Connection header received
0621 06:56:47.907863 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:47.907871 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:47.907952 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:47.908019 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:47.909454 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:47.910500 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:47.912474 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:47.913165 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:47.913299 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:47.914581 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:47.914590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:47.914598 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:47.914622 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:47.914664 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:47.914681 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:47.914784 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:47.915002 (+ 218us) server_negotiation.cc:299] Negotiation successful
0621 06:56:47.915096 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 06:56:47.916512 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:47.898146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:33391 (local address 127.20.120.194:48721)
0621 06:56:47.905292 (+ 7146us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:47.905338 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:47.907553 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:47.908500 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:47.908513 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:47.908613 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:47.909274 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:47.909295 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:47.910651 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:47.910656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:47.912327 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:47.912338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:47.913350 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:47.913354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:47.913533 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:47.914279 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:56:47.914282 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:47.914317 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:56:47.914389 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:47.914793 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:47.914801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:47.914806 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 06:56:47.914984 (+ 178us) client_negotiation.cc:240] Negotiation successful
0621 06:56:47.915342 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6595,"thread_start_us":6423,"threads_started":1}
I20260621 06:56:47.917798 8655 heartbeater.cc:347] Connected to a master server at 127.20.120.254:33391
I20260621 06:56:47.918095 8655 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:47.918773 8655 heartbeater.cc:511] Master 127.20.120.254:33391 requested a full tablet report, sending...
I20260621 06:56:47.920142 8338 ts_manager.cc:194] Registered new tserver with Master: 5c74410c5dee4fbba400b8ef8ac3bdc5 (127.20.120.194:36901)
I20260621 06:56:47.920912 8338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.194:48721
W20260621 06:56:48.236634 8658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:48.236977 8658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:48.237149 8658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:48.244160 8658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:48.244254 8658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:48.244426 8658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.195
I20260621 06:56:48.253234 8658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:33029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.195
--webserver_port=0
--tserver_master_addrs=127.20.120.254:33391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.195
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:48.254436 8658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:48.256170 8658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:48.268961 8664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:48.268970 8663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:48.268913 8666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:48.270138 8658 server_base.cc:1034] running on GCE node
I20260621 06:56:48.270717 8658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:48.271636 8658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:48.272869 8658 hybrid_clock.cc:648] HybridClock initialized: now 1782025008272826 us; error 51 us; skew 500 ppm
I20260621 06:56:48.273187 8658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 06:56:48.275264 8658 webserver.cc:460] Webserver started at http://127.20.120.195:45905/ using document root <none> and password file <none>
I20260621 06:56:48.275947 8658 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:48.276083 8658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:48.276391 8658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:48.279249 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/data/instance:
uuid: "0f38d4de315448dc9929aa2b7a727758"
format_stamp: "Formatted at 2026-06-21 06:56:48 on dist-test-slave-gmf2"
server_key: "a0d5127f395b0ac077794131779ef8ac"
server_key_iv: "c413def29e25cff4125edb54f3e5fd90"
server_key_version: "encryptionkey@0"
I20260621 06:56:48.280033 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance:
uuid: "0f38d4de315448dc9929aa2b7a727758"
format_stamp: "Formatted at 2026-06-21 06:56:48 on dist-test-slave-gmf2"
server_key: "a0d5127f395b0ac077794131779ef8ac"
server_key_iv: "c413def29e25cff4125edb54f3e5fd90"
server_key_version: "encryptionkey@0"
I20260621 06:56:48.285602 8658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 06:56:48.288995 8672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:48.290293 8658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:56:48.290457 8658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/wal
uuid: "0f38d4de315448dc9929aa2b7a727758"
format_stamp: "Formatted at 2026-06-21 06:56:48 on dist-test-slave-gmf2"
server_key: "a0d5127f395b0ac077794131779ef8ac"
server_key_iv: "c413def29e25cff4125edb54f3e5fd90"
server_key_version: "encryptionkey@0"
I20260621 06:56:48.290650 8658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:48.311172 8658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:48.311937 8658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:48.312232 8658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:48.313113 8658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 06:56:48.314469 8658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 06:56:48.314584 8658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:48.314725 8658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 06:56:48.314819 8658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:48.346946 8658 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.195:43727
I20260621 06:56:48.347028 8784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.195:43727 every 8 connection(s)
I20260621 06:56:48.348196 8658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/data/info.pb
I20260621 06:56:48.352895 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 8658
I20260621 06:56:48.353061 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/wal/instance
I20260621 06:56:48.353499 20963 external_mini_cluster.cc:1428] Setting key 8aff3855137120ea5d536b1b5db4d286
I20260621 06:56:48.360486 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.350674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:59679 (local address 127.20.120.254:33391)
0621 06:56:48.350844 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:48.350848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:48.352214 (+ 1366us) server_negotiation.cc:407] Connection header received
0621 06:56:48.354163 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:48.354169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:48.354253 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:48.354388 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:48.355631 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.356538 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.358048 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.358437 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.358526 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.359748 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:48.359753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:48.359755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:48.359793 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:48.359834 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:48.359848 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:48.359951 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:48.360122 (+ 171us) server_negotiation.cc:299] Negotiation successful
0621 06:56:48.360230 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 06:56:48.361400 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.351164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:33391 (local address 127.20.120.195:59679)
0621 06:56:48.352065 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:48.352103 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:48.353806 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:48.354626 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:48.354639 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:48.354760 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:56:48.355476 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:48.355498 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.356669 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.356673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:48.357871 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:48.357882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.358607 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.358611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:48.358801 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.359441 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 06:56:48.359444 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:56:48.359483 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 06:56:48.359557 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:56:48.359960 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:56:48.359967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:56:48.359972 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 06:56:48.360100 (+ 128us) client_negotiation.cc:240] Negotiation successful
0621 06:56:48.360364 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":322,"threads_started":1}
I20260621 06:56:48.362622 8785 heartbeater.cc:347] Connected to a master server at 127.20.120.254:33391
I20260621 06:56:48.362901 8785 heartbeater.cc:464] Registering TS with master...
I20260621 06:56:48.363557 8785 heartbeater.cc:511] Master 127.20.120.254:33391 requested a full tablet report, sending...
I20260621 06:56:48.364810 8338 ts_manager.cc:194] Registered new tserver with Master: 0f38d4de315448dc9929aa2b7a727758 (127.20.120.195:43727)
I20260621 06:56:48.365538 8338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.195:59679
I20260621 06:56:48.368204 20963 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 06:56:48.377460 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.372029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.20.120.254:33391)
0621 06:56:48.372199 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:48.372206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:48.372598 (+ 392us) server_negotiation.cc:407] Connection header received
0621 06:56:48.372778 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:48.372783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:48.372862 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:48.372960 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:48.373388 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.374069 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.375253 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.375622 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.375742 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.376748 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:48.376754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:48.376758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:48.376782 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:48.376808 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:48.376818 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:48.376925 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:48.377063 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 06:56:48.377156 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 06:56:48.380761 8338 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 06:56:48.387631 8338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 06:56:48.421425 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.399378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:36901 (local address 127.0.0.1:54072)
0621 06:56:48.411685 (+ 12307us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:48.411703 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:48.411833 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:48.412974 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:48.412982 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:48.413015 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:48.413613 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:48.413628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.415835 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.415844 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:48.418912 (+ 3068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:48.418926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.420436 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.420443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:48.420624 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.420685 (+ 61us) client_negotiation.cc:769] Sending connection context
0621 06:56:48.420832 (+ 147us) client_negotiation.cc:240] Negotiation successful
0621 06:56:48.421028 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12059,"mutex_wait_us":104,"thread_start_us":11201,"threads_started":1}
I20260621 06:56:48.421530 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.399358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.20.120.195:43727)
0621 06:56:48.410381 (+ 11023us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:48.410392 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:48.411238 (+ 846us) server_negotiation.cc:407] Connection header received
0621 06:56:48.411692 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:48.411700 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:48.411986 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:48.412179 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:48.413500 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.415665 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.418628 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.419990 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.420202 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.420436 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:48.420990 (+ 554us) server_negotiation.cc:299] Negotiation successful
0621 06:56:48.421147 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10822,"thread_start_us":10667,"threads_started":1}
I20260621 06:56:48.421700 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.399705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.20.120.194:36901)
0621 06:56:48.409956 (+ 10251us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:48.409966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:48.411979 (+ 2013us) server_negotiation.cc:407] Connection header received
0621 06:56:48.412086 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:48.412096 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:48.412376 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:48.412597 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:48.413830 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.415627 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.419093 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.420268 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.420436 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.420638 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:48.420877 (+ 239us) server_negotiation.cc:299] Negotiation successful
0621 06:56:48.421042 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10066,"thread_start_us":9944,"threads_started":1}
I20260621 06:56:48.422760 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.399354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.195:43727 (local address 127.0.0.1:60730)
0621 06:56:48.411072 (+ 11718us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:48.411127 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:48.411426 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:48.412578 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:48.412590 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:48.412651 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:48.413240 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:48.413259 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.415909 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.415921 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:48.418469 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:48.418481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.420563 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.420569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:48.420756 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.420789 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 06:56:48.420873 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 06:56:48.422348 (+ 1475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11398,"mutex_wait_us":1916,"thread_start_us":11964,"threads_started":1}
I20260621 06:56:48.425395 8590 tablet_service.cc:1467] Processing CreateTablet for tablet 56a44499dfa74b408f4f164b477c0e90 (DEFAULT_TABLE table=test-table [id=f522ed7b8dff44f7b33351a43fcebf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:48.426229 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.411877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43489 (local address 127.0.0.1:49928)
0621 06:56:48.416424 (+ 4547us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:48.416436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:48.416512 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:48.419577 (+ 3065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:48.419583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:48.419609 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:48.419936 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:48.419947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.421742 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.421749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:48.424288 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:48.424299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.425473 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.425479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:48.425632 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.425683 (+ 51us) client_negotiation.cc:769] Sending connection context
0621 06:56:48.425804 (+ 121us) client_negotiation.cc:240] Negotiation successful
0621 06:56:48.425957 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4440,"thread_start_us":4317,"threads_started":1}
I20260621 06:56:48.426362 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.411965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.20.120.193:43489)
0621 06:56:48.419031 (+ 7066us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:48.419041 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:48.419068 (+ 27us) server_negotiation.cc:407] Connection header received
0621 06:56:48.419160 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:48.419167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:48.419381 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:48.419569 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:48.420102 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.421575 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.424447 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.425280 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.425453 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.425643 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:48.425844 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 06:56:48.426010 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6899,"thread_start_us":6771,"threads_started":1}
I20260621 06:56:48.426395 8720 tablet_service.cc:1467] Processing CreateTablet for tablet 56a44499dfa74b408f4f164b477c0e90 (DEFAULT_TABLE table=test-table [id=f522ed7b8dff44f7b33351a43fcebf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:48.427662 8590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a44499dfa74b408f4f164b477c0e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:48.427830 8720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a44499dfa74b408f4f164b477c0e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:48.429857 8459 tablet_service.cc:1467] Processing CreateTablet for tablet 56a44499dfa74b408f4f164b477c0e90 (DEFAULT_TABLE table=test-table [id=f522ed7b8dff44f7b33351a43fcebf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 06:56:48.431200 8459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a44499dfa74b408f4f164b477c0e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:48.438951 8803 tablet_bootstrap.cc:492] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5: Bootstrap starting.
I20260621 06:56:48.441491 8804 tablet_bootstrap.cc:492] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758: Bootstrap starting.
I20260621 06:56:48.442800 8803 tablet_bootstrap.cc:654] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:48.444007 8803 log.cc:824] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:48.444669 8805 tablet_bootstrap.cc:492] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a: Bootstrap starting.
I20260621 06:56:48.444942 8804 tablet_bootstrap.cc:654] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:48.446218 8804 log.cc:824] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:48.447914 8805 tablet_bootstrap.cc:654] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:48.447916 8803 tablet_bootstrap.cc:492] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5: No bootstrap required, opened a new log
I20260621 06:56:48.448174 8803 ts_tablet_manager.cc:1397] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 06:56:48.449092 8805 log.cc:824] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:48.449254 8804 tablet_bootstrap.cc:492] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758: No bootstrap required, opened a new log
I20260621 06:56:48.449525 8804 ts_tablet_manager.cc:1397] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260621 06:56:48.458429 8803 raft_consensus.cc:348] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f38d4de315448dc9929aa2b7a727758" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43727 } } peers { permanent_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36901 } } peers { permanent_uuid: "a7dfda97ec9d4fa698dfe5778059083a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43489 } }
I20260621 06:56:48.459470 8803 raft_consensus.cc:374] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:48.459609 8803 raft_consensus.cc:729] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c74410c5dee4fbba400b8ef8ac3bdc5, State: Initialized, Role: FOLLOWER
I20260621 06:56:48.459694 8804 raft_consensus.cc:348] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f38d4de315448dc9929aa2b7a727758" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43727 } } peers { permanent_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36901 } } peers { permanent_uuid: "a7dfda97ec9d4fa698dfe5778059083a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43489 } }
I20260621 06:56:48.460192 8804 raft_consensus.cc:374] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:48.460323 8804 raft_consensus.cc:729] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f38d4de315448dc9929aa2b7a727758, State: Initialized, Role: FOLLOWER
I20260621 06:56:48.460178 8803 consensus_queue.cc:260] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f38d4de315448dc9929aa2b7a727758" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43727 } } peers { permanent_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36901 } } peers { permanent_uuid: "a7dfda97ec9d4fa698dfe5778059083a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43489 } }
I20260621 06:56:48.461099 8804 consensus_queue.cc:260] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f38d4de315448dc9929aa2b7a727758" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43727 } } peers { permanent_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36901 } } peers { permanent_uuid: "a7dfda97ec9d4fa698dfe5778059083a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43489 } }
I20260621 06:56:48.461874 8805 tablet_bootstrap.cc:492] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a: No bootstrap required, opened a new log
I20260621 06:56:48.462303 8805 ts_tablet_manager.cc:1397] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
I20260621 06:56:48.467103 8804 ts_tablet_manager.cc:1428] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
I20260621 06:56:48.467434 8785 heartbeater.cc:503] Master 127.20.120.254:33391 was elected leader, sending a full tablet report...
I20260621 06:56:48.469249 8655 heartbeater.cc:503] Master 127.20.120.254:33391 was elected leader, sending a full tablet report...
I20260621 06:56:48.470047 8803 ts_tablet_manager.cc:1428] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5: Time spent starting tablet: real 0.022s user 0.015s sys 0.005s
I20260621 06:56:48.474085 8805 raft_consensus.cc:348] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f38d4de315448dc9929aa2b7a727758" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43727 } } peers { permanent_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36901 } } peers { permanent_uuid: "a7dfda97ec9d4fa698dfe5778059083a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43489 } }
I20260621 06:56:48.474417 8805 raft_consensus.cc:374] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:48.474503 8805 raft_consensus.cc:729] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7dfda97ec9d4fa698dfe5778059083a, State: Initialized, Role: FOLLOWER
I20260621 06:56:48.474709 8524 heartbeater.cc:503] Master 127.20.120.254:33391 was elected leader, sending a full tablet report...
I20260621 06:56:48.475051 8805 consensus_queue.cc:260] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f38d4de315448dc9929aa2b7a727758" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43727 } } peers { permanent_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36901 } } peers { permanent_uuid: "a7dfda97ec9d4fa698dfe5778059083a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43489 } }
I20260621 06:56:48.476466 8805 ts_tablet_manager.cc:1428] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260621 06:56:48.589628 8810 raft_consensus.cc:482] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 06:56:48.589989 8810 raft_consensus.cc:504] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f38d4de315448dc9929aa2b7a727758" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43727 } } peers { permanent_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36901 } } peers { permanent_uuid: "a7dfda97ec9d4fa698dfe5778059083a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43489 } }
I20260621 06:56:48.591647 8810 leader_election.cc:288] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c74410c5dee4fbba400b8ef8ac3bdc5 (127.20.120.194:36901), a7dfda97ec9d4fa698dfe5778059083a (127.20.120.193:43489)
I20260621 06:56:48.597247 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.592276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:55243 (local address 127.20.120.193:43489)
0621 06:56:48.592516 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:48.592523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:48.592544 (+ 21us) server_negotiation.cc:407] Connection header received
0621 06:56:48.592624 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:48.592629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:48.592729 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:48.592841 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:48.593231 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.593961 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.595647 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.596448 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.596588 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.596655 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:48.596898 (+ 243us) server_negotiation.cc:299] Negotiation successful
0621 06:56:48.597008 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 06:56:48.597277 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.591928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43489 (local address 127.20.120.195:55243)
0621 06:56:48.592109 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:48.592130 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:48.592265 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:48.592900 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:48.592905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:48.592932 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:48.593087 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:48.593095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.594088 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.594093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:48.595511 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:48.595522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.596580 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.596585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:48.596751 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.596779 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 06:56:48.596873 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 06:56:48.596996 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260621 06:56:48.597575 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.592160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.120.195:36593 (local address 127.20.120.194:36901)
0621 06:56:48.592376 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:48.592382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:48.592823 (+ 441us) server_negotiation.cc:407] Connection header received
0621 06:56:48.593024 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:48.593028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:48.593092 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:48.593186 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 06:56:48.593658 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.594385 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.596062 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.596804 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.596950 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.597009 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:48.597240 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 06:56:48.597357 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 06:56:48.597733 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.592075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.194:36901 (local address 127.20.120.195:36593)
0621 06:56:48.592722 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:48.592735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 06:56:48.592885 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:56:48.593199 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:56:48.593203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:56:48.593224 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 06:56:48.593524 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:48.593532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.594524 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.594532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:48.595911 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:56:48.595921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.596949 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.596955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:56:48.597103 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.597124 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 06:56:48.597216 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 06:56:48.597326 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":2,"thread_start_us":409,"threads_started":1}
I20260621 06:56:48.598345 8479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a44499dfa74b408f4f164b477c0e90" candidate_uuid: "0f38d4de315448dc9929aa2b7a727758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7dfda97ec9d4fa698dfe5778059083a" is_pre_election: true
I20260621 06:56:48.598651 8610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a44499dfa74b408f4f164b477c0e90" candidate_uuid: "0f38d4de315448dc9929aa2b7a727758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5" is_pre_election: true
I20260621 06:56:48.598819 8479 raft_consensus.cc:2454] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f38d4de315448dc9929aa2b7a727758 in term 0.
I20260621 06:56:48.599118 8610 raft_consensus.cc:2454] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f38d4de315448dc9929aa2b7a727758 in term 0.
I20260621 06:56:48.599468 8675 leader_election.cc:302] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f38d4de315448dc9929aa2b7a727758, a7dfda97ec9d4fa698dfe5778059083a; no voters:
W20260621 06:56:48.599890 8786 tablet.cc:2369] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:56:48.599913 8810 raft_consensus.cc:2781] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 06:56:48.600049 8810 raft_consensus.cc:482] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 06:56:48.600147 8810 raft_consensus.cc:3037] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:48.602365 8810 raft_consensus.cc:504] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f38d4de315448dc9929aa2b7a727758" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43727 } } peers { permanent_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36901 } } peers { permanent_uuid: "a7dfda97ec9d4fa698dfe5778059083a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43489 } }
I20260621 06:56:48.602988 8810 leader_election.cc:288] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [CANDIDATE]: Term 1 election: Requested vote from peers 5c74410c5dee4fbba400b8ef8ac3bdc5 (127.20.120.194:36901), a7dfda97ec9d4fa698dfe5778059083a (127.20.120.193:43489)
I20260621 06:56:48.603387 8610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a44499dfa74b408f4f164b477c0e90" candidate_uuid: "0f38d4de315448dc9929aa2b7a727758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5"
I20260621 06:56:48.603519 8479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a44499dfa74b408f4f164b477c0e90" candidate_uuid: "0f38d4de315448dc9929aa2b7a727758" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7dfda97ec9d4fa698dfe5778059083a"
I20260621 06:56:48.603572 8610 raft_consensus.cc:3037] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:48.603715 8479 raft_consensus.cc:3037] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:48.605716 8610 raft_consensus.cc:2454] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f38d4de315448dc9929aa2b7a727758 in term 1.
I20260621 06:56:48.605800 8479 raft_consensus.cc:2454] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f38d4de315448dc9929aa2b7a727758 in term 1.
I20260621 06:56:48.606094 8676 leader_election.cc:302] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f38d4de315448dc9929aa2b7a727758, 5c74410c5dee4fbba400b8ef8ac3bdc5; no voters:
I20260621 06:56:48.606401 8810 raft_consensus.cc:2781] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:48.606817 8810 raft_consensus.cc:686] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 1 LEADER]: Becoming Leader. State: Replica: 0f38d4de315448dc9929aa2b7a727758, State: Running, Role: LEADER
I20260621 06:56:48.607255 8810 consensus_queue.cc:237] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f38d4de315448dc9929aa2b7a727758" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43727 } } peers { permanent_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36901 } } peers { permanent_uuid: "a7dfda97ec9d4fa698dfe5778059083a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43489 } }
I20260621 06:56:48.611505 8338 catalog_manager.cc:5495] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f38d4de315448dc9929aa2b7a727758 (127.20.120.195). New cstate: current_term: 1 leader_uuid: "0f38d4de315448dc9929aa2b7a727758" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f38d4de315448dc9929aa2b7a727758" member_type: VOTER last_known_addr { host: "127.20.120.195" port: 43727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7dfda97ec9d4fa698dfe5778059083a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43489 } health_report { overall_health: UNKNOWN } } }
W20260621 06:56:48.647163 8656 tablet.cc:2369] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:56:48.660365 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.655483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.20.120.195:43727)
0621 06:56:48.655644 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:48.655648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:48.655663 (+ 15us) server_negotiation.cc:407] Connection header received
0621 06:56:48.655745 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:48.655749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:48.655841 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:48.655963 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:56:48.656264 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.656976 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.658358 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.658749 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.658882 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.659344 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:56:48.659807 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:56:48.659972 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:48.660027 (+ 55us) server_negotiation.cc:299] Negotiation successful
0621 06:56:48.660099 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 06:56:48.667905 8610 raft_consensus.cc:1261] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5 [term 1 FOLLOWER]: Refusing update from remote peer 0f38d4de315448dc9929aa2b7a727758: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:56:48.667913 8479 raft_consensus.cc:1261] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a [term 1 FOLLOWER]: Refusing update from remote peer 0f38d4de315448dc9929aa2b7a727758: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:56:48.668784 8810 consensus_queue.cc:1035] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c74410c5dee4fbba400b8ef8ac3bdc5" member_type: VOTER last_known_addr { host: "127.20.120.194" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:56:48.669050 8813 consensus_queue.cc:1035] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7dfda97ec9d4fa698dfe5778059083a" member_type: VOTER last_known_addr { host: "127.20.120.193" port: 43489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:56:48.680500 8816 mvcc.cc:205] Tried to move back new op lower bound from 7299174435494780928 to 7299174435258929152. Current Snapshot: MvccSnapshot[applied={T|T < 7299174435494780928}]
I20260621 06:56:48.695515 8337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:56:48.695794 8337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 06:56:48.699363 8337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 638524c1674541a0a821083c14177d98: Sending DeleteTablet for 3 replicas of tablet 56a44499dfa74b408f4f164b477c0e90
I20260621 06:56:48.700464 8720 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a44499dfa74b408f4f164b477c0e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:48 UTC) from {username='slave'} at 127.0.0.1:60730
I20260621 06:56:48.700518 8590 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a44499dfa74b408f4f164b477c0e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:48 UTC) from {username='slave'} at 127.0.0.1:54072
I20260621 06:56:48.700706 8459 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a44499dfa74b408f4f164b477c0e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 06:56:48 UTC) from {username='slave'} at 127.0.0.1:49928
I20260621 06:56:48.701532 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 8397
I20260621 06:56:48.713337 8827 tablet_replica.cc:330] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a: stopping tablet replica
I20260621 06:56:48.714063 8827 raft_consensus.cc:2229] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:56:48.714833 8827 raft_consensus.cc:2258] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:48.718796 8827 ts_tablet_manager.cc:1905] T 56a44499dfa74b408f4f164b477c0e90 P a7dfda97ec9d4fa698dfe5778059083a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:48.721176 8825 tablet_replica.cc:330] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5: stopping tablet replica
I20260621 06:56:48.721961 8825 raft_consensus.cc:2229] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 06:56:48.723922 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.703191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.20.120.193:43489)
0621 06:56:48.703404 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:48.703413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:48.712957 (+ 9544us) server_negotiation.cc:407] Connection header received
0621 06:56:48.713380 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:48.713386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:48.713501 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:48.713583 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:48.714503 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.716162 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.719085 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.719871 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.719967 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.722254 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:48.722265 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:48.722281 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:48.723019 (+ 738us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:48.723076 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:48.723100 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:48.723202 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:48.723445 (+ 243us) server_negotiation.cc:299] Negotiation successful
0621 06:56:48.723555 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 06:56:48.724684 8826 tablet_replica.cc:330] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758: stopping tablet replica
I20260621 06:56:48.725190 8825 raft_consensus.cc:2258] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:48.725476 8519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:48.725088 8826 raft_consensus.cc:2229] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 1 LEADER]: Raft consensus shutting down.
I20260621 06:56:48.726547 8826 raft_consensus.cc:2258] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 06:56:48.727711 8825 ts_tablet_manager.cc:1905] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:48.728732 8826 ts_tablet_manager.cc:1905] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 06:56:48.736865 8826 ts_tablet_manager.cc:1918] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:48.736967 8825 ts_tablet_manager.cc:1918] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 06:56:48.737138 8826 log.cc:1196] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-2/wal/wals/56a44499dfa74b408f4f164b477c0e90
I20260621 06:56:48.737255 8825 log.cc:1196] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5: Deleting WAL directory at /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782024700637160-20963-0/minicluster-data/ts-1/wal/wals/56a44499dfa74b408f4f164b477c0e90
I20260621 06:56:48.737949 8826 ts_tablet_manager.cc:1939] T 56a44499dfa74b408f4f164b477c0e90 P 0f38d4de315448dc9929aa2b7a727758: Deleting consensus metadata
I20260621 06:56:48.737939 8825 ts_tablet_manager.cc:1939] T 56a44499dfa74b408f4f164b477c0e90 P 5c74410c5dee4fbba400b8ef8ac3bdc5: Deleting consensus metadata
I20260621 06:56:48.739925 8324 catalog_manager.cc:4841] TS 0f38d4de315448dc9929aa2b7a727758 (127.20.120.195:43727): tablet 56a44499dfa74b408f4f164b477c0e90 (table test-table [id=f522ed7b8dff44f7b33351a43fcebf14]) successfully deleted
I20260621 06:56:48.740268 8325 catalog_manager.cc:4841] TS 5c74410c5dee4fbba400b8ef8ac3bdc5 (127.20.120.194:36901): tablet 56a44499dfa74b408f4f164b477c0e90 (table test-table [id=f522ed7b8dff44f7b33351a43fcebf14]) successfully deleted
I20260621 06:56:48.819022 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 8397
W20260621 06:56:48.836511 8324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.120.193:43489 (error 108)
I20260621 06:56:48.837242 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 8528
I20260621 06:56:48.838768 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.838187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43489 (local address 127.0.0.1:49952)
0621 06:56:48.838461 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:48.838594 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260621 06:56:48.839129 8324 catalog_manager.cc:4584] TS a7dfda97ec9d4fa698dfe5778059083a (127.20.120.193:43489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56a44499dfa74b408f4f164b477c0e90: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43489: connect: Connection refused (error 111)
I20260621 06:56:48.848382 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.838198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.20.120.194:36901)
0621 06:56:48.838624 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:48.838628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:48.838642 (+ 14us) server_negotiation.cc:407] Connection header received
0621 06:56:48.838689 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:48.838693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:48.838771 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:48.838886 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:48.839291 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.840000 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.841237 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.841664 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.841791 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.846549 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:48.846562 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:48.846580 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:48.847500 (+ 920us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:48.847570 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:48.847600 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:48.847735 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:48.847845 (+ 110us) server_negotiation.cc:299] Negotiation successful
0621 06:56:48.847976 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321}
I20260621 06:56:48.849079 8650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:48.893025 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.892432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43489 (local address 127.0.0.1:49956)
0621 06:56:48.892701 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:48.892813 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260621 06:56:48.894284 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.893800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43489 (local address 127.0.0.1:49962)
0621 06:56:48.894015 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:48.894123 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260621 06:56:48.951380 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 8528
I20260621 06:56:48.958441 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.954295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43489 (local address 127.0.0.1:49964)
0621 06:56:48.958208 (+ 3913us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:48.958295 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3783}
I20260621 06:56:48.959543 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.959205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43489 (local address 127.0.0.1:49968)
0621 06:56:48.959365 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:48.959440 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 06:56:48.974918 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 8658
I20260621 06:56:48.983825 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:48.976350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.20.120.195:43727)
0621 06:56:48.976611 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:48.976617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:48.976667 (+ 50us) server_negotiation.cc:407] Connection header received
0621 06:56:48.976750 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:48.976756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:48.976875 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:48.976987 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:48.977535 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.978426 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.980415 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:48.980872 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:48.980974 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:48.982288 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:48.982298 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:48.982313 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:48.982991 (+ 678us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:48.983038 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:48.983058 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:48.983184 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:48.983305 (+ 121us) server_negotiation.cc:299] Negotiation successful
0621 06:56:48.983410 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260621 06:56:48.984586 8780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:49.040096 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:49.039575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43489 (local address 127.0.0.1:49980)
0621 06:56:49.039797 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:49.039900 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260621 06:56:49.041257 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:49.040875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.193:43489 (local address 127.0.0.1:49994)
0621 06:56:49.041060 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 06:56:49.041145 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.193:43489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260621 06:56:49.102547 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 8658
I20260621 06:56:49.121480 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 8307
I20260621 06:56:49.122617 8368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:49.230721 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 8307
2026-06-21T06:56:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2831 ms)
[----------] 6 tests from EncryptionPolicyTest (21220 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:56:49 dist-test-slave-gmf2 krb5kdc[8836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:56:49 dist-test-slave-gmf2 krb5kdc[8836](info): set up 2 sockets
Jun 21 06:56:49 dist-test-slave-gmf2 krb5kdc[8836](info): commencing operation
krb5kdc: starting...
W20260621 06:56:50.334389 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:56:50 dist-test-slave-gmf2 krb5kdc[8836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:56:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 06:56:50.589193 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:50.911535 8849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:50.911793 8849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:50.911967 8849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:50.918937 8849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:56:50.919027 8849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:50.919138 8849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:50.919240 8849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:56:50.919306 8849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:56:50.928366 8849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:50.929579 8849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:50.930941 8849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:50.942950 8855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:50.942929 8857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:50.943117 8854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:50.944038 8849 server_base.cc:1034] running on GCE node
I20260621 06:56:50.944818 8849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:50.946192 8849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:50.947386 8849 hybrid_clock.cc:648] HybridClock initialized: now 1782025010947328 us; error 49 us; skew 500 ppm
I20260621 06:56:50.947686 8849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:50 dist-test-slave-gmf2 krb5kdc[8836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:50.950791 8849 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 06:56:50.952222 8849 webserver.cc:460] Webserver started at http://127.0.0.1:45783/ using document root <none> and password file <none>
I20260621 06:56:50.952911 8849 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:50.953063 8849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:50.953365 8849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:50.956492 8849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "e8ecc664abef453997b772acc9a89986"
format_stamp: "Formatted at 2026-06-21 06:56:50 on dist-test-slave-gmf2"
server_key: "17c2f6b86d9899238a15e3a0bcaef88a"
server_key_iv: "9f89a71e9954d1bb9f78fee89669afd0"
server_key_version: "encryptionkey@0"
I20260621 06:56:50.957336 8849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "e8ecc664abef453997b772acc9a89986"
format_stamp: "Formatted at 2026-06-21 06:56:50 on dist-test-slave-gmf2"
server_key: "17c2f6b86d9899238a15e3a0bcaef88a"
server_key_iv: "9f89a71e9954d1bb9f78fee89669afd0"
server_key_version: "encryptionkey@0"
I20260621 06:56:50.963035 8849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260621 06:56:50.966516 8864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:50.967794 8849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:56:50.967970 8849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "e8ecc664abef453997b772acc9a89986"
format_stamp: "Formatted at 2026-06-21 06:56:50 on dist-test-slave-gmf2"
server_key: "17c2f6b86d9899238a15e3a0bcaef88a"
server_key_iv: "9f89a71e9954d1bb9f78fee89669afd0"
server_key_version: "encryptionkey@0"
I20260621 06:56:50.968170 8849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:50.988782 8849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:50.992861 8849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:50.993151 8849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:51.011735 8849 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36943
I20260621 06:56:51.011783 8915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36943 every 8 connection(s)
I20260621 06:56:51.013118 8849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:56:51.016791 8916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:51.023428 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 8849
I20260621 06:56:51.023638 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:56:51.024271 20963 external_mini_cluster.cc:1428] Setting key 3de8dc9247b2b309a03fc98a9684d2a0
I20260621 06:56:51.027122 8916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986: Bootstrap starting.
I20260621 06:56:51.031081 8916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:51.032251 8916 log.cc:824] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:51.035177 8916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986: No bootstrap required, opened a new log
Jun 21 06:56:51 dist-test-slave-gmf2 krb5kdc[8836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 06:56:51.045038 8916 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ecc664abef453997b772acc9a89986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36943 } }
I20260621 06:56:51.045396 8916 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:51.045529 8916 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ecc664abef453997b772acc9a89986, State: Initialized, Role: FOLLOWER
I20260621 06:56:51.046238 8916 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ecc664abef453997b772acc9a89986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36943 } }
I20260621 06:56:51.046491 8916 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:56:51.046639 8916 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:56:51.046854 8916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:51.049113 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:51.026204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58537 (local address 127.0.0.1:36943)
0621 06:56:51.027021 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:51.027036 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:51.027083 (+ 47us) server_negotiation.cc:407] Connection header received
0621 06:56:51.029466 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:51.029493 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:51.030039 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:51.030471 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:51.031879 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:51.033659 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:51.035181 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:51.036213 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:51.036411 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:51.040673 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:51.040700 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:51.040719 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:51.040766 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:51.044769 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:51.045497 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:51.045503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:51.045509 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:51.045715 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:51.046254 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:51.046257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:51.046259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:51.046929 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:51.047286 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:51.047798 (+ 512us) server_negotiation.cc:299] Negotiation successful
0621 06:56:51.048105 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":362,"threads_started":1}
I20260621 06:56:51.049419 8916 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ecc664abef453997b772acc9a89986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36943 } }
I20260621 06:56:51.049942 8916 leader_election.cc:302] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8ecc664abef453997b772acc9a89986; no voters:
I20260621 06:56:51.050580 8916 leader_election.cc:288] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:56:51.050724 8921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:51.052806 8921 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [term 1 LEADER]: Becoming Leader. State: Replica: e8ecc664abef453997b772acc9a89986, State: Running, Role: LEADER
I20260621 06:56:51.052901 8916 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:56:51.053475 8921 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ecc664abef453997b772acc9a89986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36943 } }
I20260621 06:56:51.053764 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:56:51.056370 8923 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8ecc664abef453997b772acc9a89986. Latest consensus state: current_term: 1 leader_uuid: "e8ecc664abef453997b772acc9a89986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ecc664abef453997b772acc9a89986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36943 } } }
I20260621 06:56:51.056398 8922 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8ecc664abef453997b772acc9a89986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8ecc664abef453997b772acc9a89986" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36943 } } }
I20260621 06:56:51.056846 8923 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:51.056846 8922 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:51.063078 8927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:56:51.068097 8927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:56:51.084631 8927 catalog_manager.cc:1347] Generated new cluster ID: 9fd5c94cba41416ca2189c50510add1d
I20260621 06:56:51.084750 8927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:56:51.099246 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:51.074372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32923 (local address 127.0.0.1:36943)
0621 06:56:51.074539 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:51.074545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:51.082018 (+ 7473us) server_negotiation.cc:407] Connection header received
0621 06:56:51.082834 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:51.082839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:51.082937 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:51.083054 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:51.085218 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:51.086422 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:51.088155 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:51.088769 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:51.088858 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:51.092307 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:51.092313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:51.092318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:51.092363 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:51.095878 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:51.096818 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:51.096824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:51.096830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:51.096979 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:51.097455 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:51.097459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:51.097463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:51.097813 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:51.098048 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:51.098708 (+ 660us) server_negotiation.cc:299] Negotiation successful
0621 06:56:51.098892 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 06:56:51.111907 8927 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:56:51.113236 8927 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:56:51.123544 8927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8ecc664abef453997b772acc9a89986: Generated new TSK 0
I20260621 06:56:51.124421 8927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:56:51.131397 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 8849
I20260621 06:56:51.133112 8911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:51.228957 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 8849
W20260621 06:56:51.246304 8834 connection.cc:535] client connection to 127.0.0.1:36943 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36943): Connection reset by peer (error 104)
2026-06-21T06:56:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:56:51 dist-test-slave-gmf2 krb5kdc[8948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:56:51 dist-test-slave-gmf2 krb5kdc[8948](info): set up 2 sockets
Jun 21 06:56:51 dist-test-slave-gmf2 krb5kdc[8948](info): commencing operation
krb5kdc: starting...
W20260621 06:56:52.338207 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:56:52 dist-test-slave-gmf2 krb5kdc[8948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:56:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:56:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 06:56:52.571237 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:52.895747 8961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:52.896019 8961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:52.896211 8961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:52.903329 8961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:56:52.903456 8961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:52.903556 8961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:52.903664 8961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:56:52.903756 8961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:56:52.913266 8961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:52.914667 8961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:52.916025 8961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:52.930205 8966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:52.930258 8967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:52.930297 8969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:52.931802 8961 server_base.cc:1034] running on GCE node
I20260621 06:56:52.932519 8961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:52.934073 8961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:52.935272 8961 hybrid_clock.cc:648] HybridClock initialized: now 1782025012935218 us; error 62 us; skew 500 ppm
I20260621 06:56:52.935629 8961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:52 dist-test-slave-gmf2 krb5kdc[8948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:52.938823 8961 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 06:56:52.940187 8961 webserver.cc:460] Webserver started at http://127.0.0.1:37811/ using document root <none> and password file <none>
I20260621 06:56:52.940845 8961 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:52.940948 8961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:52.941195 8961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:52.944285 8961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "045091e9aa6c48c9b8b0008817642f24"
format_stamp: "Formatted at 2026-06-21 06:56:52 on dist-test-slave-gmf2"
server_key: "a25e524908d54ca90c2368226a174a22"
server_key_iv: "126c2a6d298622c78c9b787461a194d0"
server_key_version: "encryptionkey@0"
I20260621 06:56:52.945024 8961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "045091e9aa6c48c9b8b0008817642f24"
format_stamp: "Formatted at 2026-06-21 06:56:52 on dist-test-slave-gmf2"
server_key: "a25e524908d54ca90c2368226a174a22"
server_key_iv: "126c2a6d298622c78c9b787461a194d0"
server_key_version: "encryptionkey@0"
I20260621 06:56:52.950302 8961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260621 06:56:52.953817 8976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:52.955201 8961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:56:52.955391 8961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "045091e9aa6c48c9b8b0008817642f24"
format_stamp: "Formatted at 2026-06-21 06:56:52 on dist-test-slave-gmf2"
server_key: "a25e524908d54ca90c2368226a174a22"
server_key_iv: "126c2a6d298622c78c9b787461a194d0"
server_key_version: "encryptionkey@0"
I20260621 06:56:52.955576 8961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:52.970956 8961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:52.975647 8961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:52.976042 8961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:52.995042 8961 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39113
I20260621 06:56:52.995064 9027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39113 every 8 connection(s)
I20260621 06:56:52.996416 8961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:56:53.000133 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:53.004590 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 8961
I20260621 06:56:53.004791 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:56:53.005218 20963 external_mini_cluster.cc:1428] Setting key 8874786322ff668326094208403d6008
I20260621 06:56:53.012684 9028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24: Bootstrap starting.
I20260621 06:56:53.016383 9028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:53.017634 9028 log.cc:824] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:53.020967 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:53.007108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49325 (local address 127.0.0.1:39113)
0621 06:56:53.007943 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:53.007958 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:53.008007 (+ 49us) server_negotiation.cc:407] Connection header received
0621 06:56:53.010776 (+ 2769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:53.010806 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:53.011333 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:53.011777 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:53.012300 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:53.014296 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:53.015673 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:53.016768 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:53.016969 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:53.018241 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:53.018259 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:53.018286 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:53.018921 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:53.019039 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:53.019073 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:53.019233 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:53.019488 (+ 255us) server_negotiation.cc:299] Negotiation successful
0621 06:56:53.019665 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":346,"threads_started":1}
I20260621 06:56:53.021080 9028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24: No bootstrap required, opened a new log
I20260621 06:56:53.024885 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:56:53.029695 9028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045091e9aa6c48c9b8b0008817642f24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260621 06:56:53.030279 9028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:53.030390 9028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045091e9aa6c48c9b8b0008817642f24, State: Initialized, Role: FOLLOWER
I20260621 06:56:53.031128 9028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045091e9aa6c48c9b8b0008817642f24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260621 06:56:53.031414 9028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:56:53.031574 9028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:56:53.031782 9028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:53.034138 9028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045091e9aa6c48c9b8b0008817642f24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260621 06:56:53.034759 9028 leader_election.cc:302] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 045091e9aa6c48c9b8b0008817642f24; no voters:
I20260621 06:56:53.035507 9028 leader_election.cc:288] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:56:53.035624 9038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:53.036140 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:53.029248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53885 (local address 127.0.0.1:39113)
0621 06:56:53.029468 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:53.029477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:53.029994 (+ 517us) server_negotiation.cc:407] Connection header received
0621 06:56:53.030281 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:53.030285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:53.030349 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:53.030478 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:53.031012 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:53.032061 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:53.033352 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:53.033819 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:53.033992 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:53.035213 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:53.035218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:53.035222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:53.035244 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:53.035302 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:53.035322 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:53.035462 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:53.035669 (+ 207us) server_negotiation.cc:299] Negotiation successful
0621 06:56:53.035735 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 06:56:53.036302 9038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [term 1 LEADER]: Becoming Leader. State: Replica: 045091e9aa6c48c9b8b0008817642f24, State: Running, Role: LEADER
I20260621 06:56:53.037143 9038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045091e9aa6c48c9b8b0008817642f24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260621 06:56:53.040038 9028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:56:53.040438 9040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "045091e9aa6c48c9b8b0008817642f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045091e9aa6c48c9b8b0008817642f24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } } }
I20260621 06:56:53.040408 9039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 045091e9aa6c48c9b8b0008817642f24. Latest consensus state: current_term: 1 leader_uuid: "045091e9aa6c48c9b8b0008817642f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045091e9aa6c48c9b8b0008817642f24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } } }
I20260621 06:56:53.040834 9039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:53.040831 9040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:53.042629 9042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:56:53.052613 9042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:56:53.061729 9042 catalog_manager.cc:1347] Generated new cluster ID: fc36a2cc93f24ca2b86dc20cf0e062ca
I20260621 06:56:53.061944 9042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:56:53.105123 9042 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:56:53.107127 9042 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:56:53.115829 9042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 045091e9aa6c48c9b8b0008817642f24: Generated new TSK 0
I20260621 06:56:53.116779 9042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:56:53.124352 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 8961
W20260621 06:56:53.126339 9023 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49325
W20260621 06:56:53.127171 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 8961: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:56:53.127357 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 8961
W20260621 06:56:53.144495 8946 connection.cc:535] client connection to 127.0.0.1:39113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39113): Connection reset by peer (error 104)
2026-06-21T06:56:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:56:53 dist-test-slave-gmf2 krb5kdc[9059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:56:53 dist-test-slave-gmf2 krb5kdc[9059](info): set up 2 sockets
Jun 21 06:56:53 dist-test-slave-gmf2 krb5kdc[9059](info): commencing operation
krb5kdc: starting...
W20260621 06:56:54.210220 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:56:54 dist-test-slave-gmf2 krb5kdc[9059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:56:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:56:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 06:56:54.452944 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:54.784029 9072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:54.784312 9072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:54.784463 9072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:54.791709 9072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:56:54.791849 9072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:54.791916 9072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:54.792037 9072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:56:54.792109 9072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:56:54.802459 9072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:54.803845 9072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:54.805241 9072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:54.818043 9077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:54.817970 9080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:54.818015 9078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:54.818804 9072 server_base.cc:1034] running on GCE node
I20260621 06:56:54.819671 9072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:54.821138 9072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:54.822374 9072 hybrid_clock.cc:648] HybridClock initialized: now 1782025014822304 us; error 69 us; skew 500 ppm
I20260621 06:56:54.822729 9072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:54 dist-test-slave-gmf2 krb5kdc[9059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:54.829411 9072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 06:56:54.830935 9072 webserver.cc:460] Webserver started at http://127.0.0.1:38199/ using document root <none> and password file <none>
I20260621 06:56:54.831620 9072 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:54.831775 9072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:54.832083 9072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:54.835158 9072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "a8696b8dd865408d9e1a04b8eb7fe3d3"
format_stamp: "Formatted at 2026-06-21 06:56:54 on dist-test-slave-gmf2"
server_key: "65d52d13dc14176a179766880277c810"
server_key_iv: "dc6039c810597adbdad08fecdd9f8c1d"
server_key_version: "encryptionkey@0"
I20260621 06:56:54.836041 9072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "a8696b8dd865408d9e1a04b8eb7fe3d3"
format_stamp: "Formatted at 2026-06-21 06:56:54 on dist-test-slave-gmf2"
server_key: "65d52d13dc14176a179766880277c810"
server_key_iv: "dc6039c810597adbdad08fecdd9f8c1d"
server_key_version: "encryptionkey@0"
I20260621 06:56:54.841387 9072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260621 06:56:54.845140 9087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:54.846663 9072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:56:54.846848 9072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "a8696b8dd865408d9e1a04b8eb7fe3d3"
format_stamp: "Formatted at 2026-06-21 06:56:54 on dist-test-slave-gmf2"
server_key: "65d52d13dc14176a179766880277c810"
server_key_iv: "dc6039c810597adbdad08fecdd9f8c1d"
server_key_version: "encryptionkey@0"
I20260621 06:56:54.847059 9072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:54.852823 9072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:54.857204 9072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:54.857515 9072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:54.875519 9072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33757
I20260621 06:56:54.875559 9138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33757 every 8 connection(s)
I20260621 06:56:54.876950 9072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:56:54.880510 9139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:54.886935 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 9072
I20260621 06:56:54.887113 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:56:54.887516 20963 external_mini_cluster.cc:1428] Setting key 4fff0739f63e3d403dbd4ca2285de23a
I20260621 06:56:54.891136 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3: Bootstrap starting.
I20260621 06:56:54.894418 9139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:54.895387 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:54.889365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56639 (local address 127.0.0.1:33757)
0621 06:56:54.890185 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:54.890197 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:54.890235 (+ 38us) server_negotiation.cc:407] Connection header received
0621 06:56:54.891959 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:54.891980 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:54.892318 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:54.892713 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:54.893764 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:54.893770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:54.893783 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:54.893797 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:54.893857 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:54.893876 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:54.894157 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:54.894290 (+ 133us) server_negotiation.cc:299] Negotiation successful
0621 06:56:54.894470 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":346,"threads_started":1}
I20260621 06:56:54.895532 9139 log.cc:824] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:54.898612 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3: No bootstrap required, opened a new log
I20260621 06:56:54.899091 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:56:54.905721 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:54.903302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45689 (local address 127.0.0.1:33757)
0621 06:56:54.903526 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:54.903535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:54.903887 (+ 352us) server_negotiation.cc:407] Connection header received
0621 06:56:54.904046 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:54.904050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:54.904102 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:54.904193 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:54.905061 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:54.905065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:54.905068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:54.905084 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:54.905120 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:54.905133 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:54.905241 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:54.905366 (+ 125us) server_negotiation.cc:299] Negotiation successful
0621 06:56:54.905414 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 06:56:54.907024 9139 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8696b8dd865408d9e1a04b8eb7fe3d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33757 } }
I20260621 06:56:54.907403 9139 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:54.907518 9139 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8696b8dd865408d9e1a04b8eb7fe3d3, State: Initialized, Role: FOLLOWER
I20260621 06:56:54.908084 9139 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8696b8dd865408d9e1a04b8eb7fe3d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33757 } }
I20260621 06:56:54.908342 9139 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:56:54.908491 9139 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:56:54.908684 9139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:54.911393 9139 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8696b8dd865408d9e1a04b8eb7fe3d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33757 } }
I20260621 06:56:54.911978 9139 leader_election.cc:302] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8696b8dd865408d9e1a04b8eb7fe3d3; no voters:
I20260621 06:56:54.912602 9139 leader_election.cc:288] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:56:54.912703 9149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:54.913430 9149 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [term 1 LEADER]: Becoming Leader. State: Replica: a8696b8dd865408d9e1a04b8eb7fe3d3, State: Running, Role: LEADER
I20260621 06:56:54.914079 9149 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8696b8dd865408d9e1a04b8eb7fe3d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33757 } }
I20260621 06:56:54.914971 9139 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:56:54.916913 9150 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8696b8dd865408d9e1a04b8eb7fe3d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8696b8dd865408d9e1a04b8eb7fe3d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33757 } } }
I20260621 06:56:54.916923 9151 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8696b8dd865408d9e1a04b8eb7fe3d3. Latest consensus state: current_term: 1 leader_uuid: "a8696b8dd865408d9e1a04b8eb7fe3d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8696b8dd865408d9e1a04b8eb7fe3d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33757 } } }
I20260621 06:56:54.917441 9151 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:54.917449 9150 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:54.921422 9154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:56:54.926788 9154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:56:54.935909 9154 catalog_manager.cc:1347] Generated new cluster ID: bd120d81d1714a9da01b3196e0822461
I20260621 06:56:54.936043 9154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:56:54.945223 9154 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:56:54.945674 9154 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:56:54.951051 9154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8696b8dd865408d9e1a04b8eb7fe3d3: Generated new TSK 0
I20260621 06:56:54.951970 9154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:56:54.957338 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9072
W20260621 06:56:54.958472 9134 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56639
W20260621 06:56:54.959156 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9072: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:56:54.959287 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 9072
2026-06-21T06:56:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1828 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:56:55 dist-test-slave-gmf2 krb5kdc[9170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:56:55 dist-test-slave-gmf2 krb5kdc[9170](info): set up 2 sockets
Jun 21 06:56:55 dist-test-slave-gmf2 krb5kdc[9170](info): commencing operation
krb5kdc: starting...
W20260621 06:56:56.066459 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:56:56 dist-test-slave-gmf2 krb5kdc[9170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:56:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:56:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 06:56:56.296957 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:56.613188 9183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:56.613539 9183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:56.613729 9183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:56.621097 9183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:56:56.621268 9183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:56.621377 9183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:56.621474 9183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:56:56.621562 9183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:56:56.631489 9183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:56.632787 9183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:56.634120 9183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:56.645880 9188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:56.645855 9189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:56.646600 9191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:56.647030 9183 server_base.cc:1034] running on GCE node
I20260621 06:56:56.647781 9183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:56.649173 9183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:56.650398 9183 hybrid_clock.cc:648] HybridClock initialized: now 1782025016650355 us; error 55 us; skew 500 ppm
I20260621 06:56:56.650775 9183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:56 dist-test-slave-gmf2 krb5kdc[9170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:56.653983 9183 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 06:56:56.655424 9183 webserver.cc:460] Webserver started at http://127.0.0.1:33561/ using document root <none> and password file <none>
I20260621 06:56:56.656108 9183 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:56.656252 9183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:56.656551 9183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:56.659576 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "59a3e1cac05d42359a8b7280cc8b90ae"
format_stamp: "Formatted at 2026-06-21 06:56:56 on dist-test-slave-gmf2"
server_key: "5520c1f11883bdfda065ebd6714c6cb3"
server_key_iv: "92977d285a349e2eb21aeeec8f0b0849"
server_key_version: "encryptionkey@0"
I20260621 06:56:56.660378 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "59a3e1cac05d42359a8b7280cc8b90ae"
format_stamp: "Formatted at 2026-06-21 06:56:56 on dist-test-slave-gmf2"
server_key: "5520c1f11883bdfda065ebd6714c6cb3"
server_key_iv: "92977d285a349e2eb21aeeec8f0b0849"
server_key_version: "encryptionkey@0"
I20260621 06:56:56.665356 9183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 06:56:56.668895 9198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:56.670286 9183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:56:56.670469 9183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "59a3e1cac05d42359a8b7280cc8b90ae"
format_stamp: "Formatted at 2026-06-21 06:56:56 on dist-test-slave-gmf2"
server_key: "5520c1f11883bdfda065ebd6714c6cb3"
server_key_iv: "92977d285a349e2eb21aeeec8f0b0849"
server_key_version: "encryptionkey@0"
I20260621 06:56:56.670665 9183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:56.685089 9183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:56.689265 9183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:56.689545 9183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:56.709095 9183 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43975
I20260621 06:56:56.709110 9249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43975 every 8 connection(s)
I20260621 06:56:56.710661 9183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:56:56.714474 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:56.719952 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 9183
I20260621 06:56:56.720132 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:56:56.720660 20963 external_mini_cluster.cc:1428] Setting key 7f0aebdb32a997d78a4fc1fc5b664699
I20260621 06:56:56.725626 9250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae: Bootstrap starting.
I20260621 06:56:56.729285 9250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:56.730562 9250 log.cc:824] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:56.733956 9250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae: No bootstrap required, opened a new log
Jun 21 06:56:56 dist-test-slave-gmf2 krb5kdc[9170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 06:56:56.744422 9250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a3e1cac05d42359a8b7280cc8b90ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43975 } }
I20260621 06:56:56.744877 9250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:56.744984 9250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a3e1cac05d42359a8b7280cc8b90ae, State: Initialized, Role: FOLLOWER
I20260621 06:56:56.745437 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:56.722473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45779 (local address 127.0.0.1:43975)
0621 06:56:56.723338 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:56.723350 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:56.723392 (+ 42us) server_negotiation.cc:407] Connection header received
0621 06:56:56.725041 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:56.725062 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:56.725440 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:56.725770 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:56.727529 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:56.728780 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:56.730714 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:56.731732 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:56.731914 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:56.736676 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:56.736686 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:56.736709 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:56.736751 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:56.740840 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:56.741616 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:56.741626 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:56.741635 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:56.741857 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:56.742248 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:56.742253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:56.742257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:56.743003 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:56.743309 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:56.743610 (+ 301us) server_negotiation.cc:299] Negotiation successful
0621 06:56:56.744048 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":414,"threads_started":1}
I20260621 06:56:56.745666 9250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a3e1cac05d42359a8b7280cc8b90ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43975 } }
I20260621 06:56:56.745980 9250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:56:56.746157 9250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:56:56.746347 9250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:56.749382 9250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a3e1cac05d42359a8b7280cc8b90ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43975 } }
I20260621 06:56:56.749522 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:56:56.749948 9250 leader_election.cc:302] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59a3e1cac05d42359a8b7280cc8b90ae; no voters:
I20260621 06:56:56.750520 9250 leader_election.cc:288] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:56:56.750646 9255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:56.751856 9255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [term 1 LEADER]: Becoming Leader. State: Replica: 59a3e1cac05d42359a8b7280cc8b90ae, State: Running, Role: LEADER
I20260621 06:56:56.752385 9255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a3e1cac05d42359a8b7280cc8b90ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43975 } }
I20260621 06:56:56.752635 9250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:56:56.755093 9257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59a3e1cac05d42359a8b7280cc8b90ae. Latest consensus state: current_term: 1 leader_uuid: "59a3e1cac05d42359a8b7280cc8b90ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a3e1cac05d42359a8b7280cc8b90ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43975 } } }
I20260621 06:56:56.755470 9257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:56.755991 9256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59a3e1cac05d42359a8b7280cc8b90ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a3e1cac05d42359a8b7280cc8b90ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43975 } } }
I20260621 06:56:56.756311 9256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:56.763995 9262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:56:56.769362 9262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:56:56.793447 9262 catalog_manager.cc:1347] Generated new cluster ID: 195d6cd402014f719c3c4bfbb8e871b2
I20260621 06:56:56.793571 9262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:56:56.800086 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:56.775790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50153 (local address 127.0.0.1:43975)
0621 06:56:56.776009 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:56.776018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:56.780803 (+ 4785us) server_negotiation.cc:407] Connection header received
0621 06:56:56.781670 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:56.781680 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:56.781790 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:56.781882 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:56.784504 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:56.786013 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:56.788489 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:56.789460 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:56.789591 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:56.793306 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:56.793313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:56.793320 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:56:56.793382 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:56.797051 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:56.797764 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:56.797770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:56.797773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:56.797909 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:56:56.798335 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:56:56.798340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:56:56.798343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:56:56.798754 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:56.799003 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:56.799330 (+ 327us) server_negotiation.cc:299] Negotiation successful
0621 06:56:56.799559 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:56:56.814797 9262 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:56:56.816730 9262 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:56:56.832798 9262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59a3e1cac05d42359a8b7280cc8b90ae: Generated new TSK 0
I20260621 06:56:56.834010 9262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:56:56.849656 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9183
I20260621 06:56:56.850770 9245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:56:56.940254 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 9183
2026-06-21T06:56:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:56:57 dist-test-slave-gmf2 krb5kdc[9283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:56:57 dist-test-slave-gmf2 krb5kdc[9283](info): set up 2 sockets
Jun 21 06:56:57 dist-test-slave-gmf2 krb5kdc[9283](info): commencing operation
krb5kdc: starting...
W20260621 06:56:58.026301 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:56:58 dist-test-slave-gmf2 krb5kdc[9283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:56:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:56:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 06:56:58.265205 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:56:58.579468 9296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:56:58.579739 9296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:56:58.579908 9296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:56:58.586696 9296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:56:58.586818 9296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:56:58.586930 9296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:56:58.587054 9296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:56:58.587143 9296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:56:58.596076 9296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:56:58.597211 9296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:56:58.598448 9296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:56:58.609755 9302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:58.609829 9301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:56:58.609803 9304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:56:58.610381 9296 server_base.cc:1034] running on GCE node
I20260621 06:56:58.611001 9296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:56:58.612166 9296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:56:58.613387 9296 hybrid_clock.cc:648] HybridClock initialized: now 1782025018613320 us; error 46 us; skew 500 ppm
I20260621 06:56:58.613708 9296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:56:58 dist-test-slave-gmf2 krb5kdc[9283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:56:58.616636 9296 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 06:56:58.617826 9296 webserver.cc:460] Webserver started at http://127.0.0.1:40259/ using document root <none> and password file <none>
I20260621 06:56:58.618440 9296 fs_manager.cc:359] Metadata directory not provided
I20260621 06:56:58.618532 9296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:56:58.618750 9296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:56:58.621488 9296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "6f2f01d8fcc64f03a205e58ebd739068"
format_stamp: "Formatted at 2026-06-21 06:56:58 on dist-test-slave-gmf2"
server_key: "08990e6328013e8fe94f6eea727e4773"
server_key_iv: "18d55b8ee7a4978d0d8bc8b82733f9e8"
server_key_version: "encryptionkey@0"
I20260621 06:56:58.622226 9296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "6f2f01d8fcc64f03a205e58ebd739068"
format_stamp: "Formatted at 2026-06-21 06:56:58 on dist-test-slave-gmf2"
server_key: "08990e6328013e8fe94f6eea727e4773"
server_key_iv: "18d55b8ee7a4978d0d8bc8b82733f9e8"
server_key_version: "encryptionkey@0"
I20260621 06:56:58.626891 9296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 06:56:58.630039 9311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:56:58.631225 9296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 06:56:58.631389 9296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "6f2f01d8fcc64f03a205e58ebd739068"
format_stamp: "Formatted at 2026-06-21 06:56:58 on dist-test-slave-gmf2"
server_key: "08990e6328013e8fe94f6eea727e4773"
server_key_iv: "18d55b8ee7a4978d0d8bc8b82733f9e8"
server_key_version: "encryptionkey@0"
I20260621 06:56:58.631564 9296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:56:58.653527 9296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:56:58.657235 9296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:56:58.657502 9296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:56:58.675776 9296 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43509
I20260621 06:56:58.675796 9362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43509 every 8 connection(s)
I20260621 06:56:58.677119 9296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:56:58.680678 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:56:58.687202 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 9296
I20260621 06:56:58.687397 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:56:58.687892 20963 external_mini_cluster.cc:1428] Setting key 22b32449022b14a5c36544c058546d59
I20260621 06:56:58.691370 9363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068: Bootstrap starting.
I20260621 06:56:58.694670 9363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068: Neither blocks nor log segments found. Creating new log.
I20260621 06:56:58.695909 9363 log.cc:824] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068: Log is configured to *not* fsync() on all Append() calls
I20260621 06:56:58.698895 9363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068: No bootstrap required, opened a new log
I20260621 06:56:58.700914 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:58.689726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49599 (local address 127.0.0.1:43509)
0621 06:56:58.690511 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:58.690522 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:58.690559 (+ 37us) server_negotiation.cc:407] Connection header received
0621 06:56:58.692272 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:58.692293 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:58.692704 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:58.693025 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:58.693579 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:58.694931 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:58.696730 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:58.697502 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:58.697653 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:58.698837 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:58.698844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:58.698855 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:58.699393 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:58.699456 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:58.699477 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:58.699559 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:58.699793 (+ 234us) server_negotiation.cc:299] Negotiation successful
0621 06:56:58.700034 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":391,"threads_started":1}
I20260621 06:56:58.704630 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:56:58.708314 9363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2f01d8fcc64f03a205e58ebd739068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } }
I20260621 06:56:58.708813 9363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:56:58.708925 9363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2f01d8fcc64f03a205e58ebd739068, State: Initialized, Role: FOLLOWER
I20260621 06:56:58.709517 9363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2f01d8fcc64f03a205e58ebd739068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } }
I20260621 06:56:58.709836 9363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:56:58.710100 9363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:56:58.710268 9363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:56:58.713178 9363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2f01d8fcc64f03a205e58ebd739068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } }
I20260621 06:56:58.713709 9363 leader_election.cc:302] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f2f01d8fcc64f03a205e58ebd739068; no voters:
I20260621 06:56:58.714342 9363 leader_election.cc:288] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:56:58.714458 9373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:56:58.715766 9373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [term 1 LEADER]: Becoming Leader. State: Replica: 6f2f01d8fcc64f03a205e58ebd739068, State: Running, Role: LEADER
I20260621 06:56:58.716418 9373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2f01d8fcc64f03a205e58ebd739068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } }
I20260621 06:56:58.716915 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:56:58.709895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42463 (local address 127.0.0.1:43509)
0621 06:56:58.710093 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 06:56:58.710100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:56:58.710517 (+ 417us) server_negotiation.cc:407] Connection header received
0621 06:56:58.710648 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:56:58.710656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:56:58.710731 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:56:58.710846 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:56:58.711303 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:58.712397 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:58.713776 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:56:58.714558 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:56:58.714711 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:56:58.716043 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:56:58.716049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:56:58.716053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:56:58.716081 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:56:58.716125 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:56:58.716149 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:56:58.716254 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 06:56:58.716454 (+ 200us) server_negotiation.cc:299] Negotiation successful
0621 06:56:58.716558 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260621 06:56:58.717015 9363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:56:58.719480 9374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f2f01d8fcc64f03a205e58ebd739068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2f01d8fcc64f03a205e58ebd739068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } } }
I20260621 06:56:58.719595 9375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f2f01d8fcc64f03a205e58ebd739068. Latest consensus state: current_term: 1 leader_uuid: "6f2f01d8fcc64f03a205e58ebd739068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2f01d8fcc64f03a205e58ebd739068" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } } }
I20260621 06:56:58.719964 9374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:58.720011 9375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068 [sys.catalog]: This master's current role is: LEADER
I20260621 06:56:58.721670 9378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:56:58.727813 9378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:56:58.736490 9378 catalog_manager.cc:1347] Generated new cluster ID: 5344fa392b784b549ea7899072f46e81
I20260621 06:56:58.736590 9378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:56:58.749754 9378 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:56:58.751013 9378 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:56:58.768203 9378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f2f01d8fcc64f03a205e58ebd739068: Generated new TSK 0
I20260621 06:56:58.769006 9378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:56:58.775022 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9296
W20260621 06:56:58.776075 9358 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49599
W20260621 06:56:58.776748 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9296: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:56:58.776893 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 9296
2026-06-21T06:56:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1835 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:56:58 dist-test-slave-gmf2 krb5kdc[9394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:56:58 dist-test-slave-gmf2 krb5kdc[9394](info): set up 2 sockets
Jun 21 06:56:58 dist-test-slave-gmf2 krb5kdc[9394](info): commencing operation
krb5kdc: starting...
W20260621 06:56:59.862473 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:00 dist-test-slave-gmf2 krb5kdc[9394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 06:57:00.125206 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:00.448364 9407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:00.448709 9407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:00.448869 9407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:00.456105 9407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:00.456277 9407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:00.456394 9407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:00.456501 9407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:00.456589 9407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:00.466754 9407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:00.468154 9407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:00.469556 9407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:00.481575 9413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:00.481703 9415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:00.481814 9412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:00.482684 9407 server_base.cc:1034] running on GCE node
I20260621 06:57:00.483446 9407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:00.484789 9407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:00.486002 9407 hybrid_clock.cc:648] HybridClock initialized: now 1782025020485933 us; error 61 us; skew 500 ppm
I20260621 06:57:00.486328 9407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:00 dist-test-slave-gmf2 krb5kdc[9394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:00.494377 9407 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 06:57:00.495811 9407 webserver.cc:460] Webserver started at http://127.0.0.1:41977/ using document root <none> and password file <none>
I20260621 06:57:00.496490 9407 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:00.496644 9407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:00.496949 9407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:00.499972 9407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "641dce09473941459a12f6d43c3edde7"
format_stamp: "Formatted at 2026-06-21 06:57:00 on dist-test-slave-gmf2"
server_key: "0694f00ab1833739acfb3299682763a8"
server_key_iv: "f543c4d7df5eb21b9fd912830cc28e2d"
server_key_version: "encryptionkey@0"
I20260621 06:57:00.500741 9407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "641dce09473941459a12f6d43c3edde7"
format_stamp: "Formatted at 2026-06-21 06:57:00 on dist-test-slave-gmf2"
server_key: "0694f00ab1833739acfb3299682763a8"
server_key_iv: "f543c4d7df5eb21b9fd912830cc28e2d"
server_key_version: "encryptionkey@0"
I20260621 06:57:00.505709 9407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260621 06:57:00.509006 9422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:00.510360 9407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:57:00.510531 9407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "641dce09473941459a12f6d43c3edde7"
format_stamp: "Formatted at 2026-06-21 06:57:00 on dist-test-slave-gmf2"
server_key: "0694f00ab1833739acfb3299682763a8"
server_key_iv: "f543c4d7df5eb21b9fd912830cc28e2d"
server_key_version: "encryptionkey@0"
I20260621 06:57:00.510753 9407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:00.516453 9407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:00.520756 9407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:00.521049 9407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:00.541646 9407 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40169
I20260621 06:57:00.541690 9473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40169 every 8 connection(s)
I20260621 06:57:00.543135 9407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:00.546927 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:00.550992 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 9407
I20260621 06:57:00.551198 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:00.551625 20963 external_mini_cluster.cc:1428] Setting key 2cbeda209ba91d1386d118b3420d4982
I20260621 06:57:00.558015 9474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7: Bootstrap starting.
I20260621 06:57:00.559744 9477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:00.553429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36223 (local address 127.0.0.1:40169)
0621 06:57:00.554252 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:00.554263 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:00.554304 (+ 41us) server_negotiation.cc:407] Connection header received
0621 06:57:00.556446 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:00.556469 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:00.556815 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:00.557140 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:00.558242 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:00.558249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:00.558274 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:00.558287 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:00.558341 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:00.558359 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:00.558513 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:00.558650 (+ 137us) server_negotiation.cc:299] Negotiation successful
0621 06:57:00.558840 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":353,"threads_started":1}
I20260621 06:57:00.561295 9474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:00.562501 9474 log.cc:824] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:00.563505 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:00.565229 9474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7: No bootstrap required, opened a new log
I20260621 06:57:00.570367 9477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:00.567939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51035 (local address 127.0.0.1:40169)
0621 06:57:00.568129 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:00.568136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:00.568386 (+ 250us) server_negotiation.cc:407] Connection header received
0621 06:57:00.568559 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:00.568563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:00.568607 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:00.568691 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:00.569625 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:00.569629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:00.569632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:00.569645 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:00.569683 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:00.569700 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:00.569790 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:00.569975 (+ 185us) server_negotiation.cc:299] Negotiation successful
0621 06:57:00.570031 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 06:57:00.573873 9474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641dce09473941459a12f6d43c3edde7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } }
I20260621 06:57:00.574350 9474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:00.574455 9474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641dce09473941459a12f6d43c3edde7, State: Initialized, Role: FOLLOWER
I20260621 06:57:00.575008 9474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641dce09473941459a12f6d43c3edde7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } }
I20260621 06:57:00.575275 9474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:00.575412 9474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:00.575587 9474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:00.578158 9474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641dce09473941459a12f6d43c3edde7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } }
I20260621 06:57:00.578658 9474 leader_election.cc:302] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 641dce09473941459a12f6d43c3edde7; no voters:
I20260621 06:57:00.579236 9474 leader_election.cc:288] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:00.579329 9484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:00.580878 9484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [term 1 LEADER]: Becoming Leader. State: Replica: 641dce09473941459a12f6d43c3edde7, State: Running, Role: LEADER
I20260621 06:57:00.581436 9484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641dce09473941459a12f6d43c3edde7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } }
I20260621 06:57:00.581610 9474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:00.584390 9486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 641dce09473941459a12f6d43c3edde7. Latest consensus state: current_term: 1 leader_uuid: "641dce09473941459a12f6d43c3edde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641dce09473941459a12f6d43c3edde7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } } }
I20260621 06:57:00.584506 9485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "641dce09473941459a12f6d43c3edde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641dce09473941459a12f6d43c3edde7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40169 } } }
I20260621 06:57:00.584813 9486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:00.584818 9485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:00.585923 9493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:00.594756 9493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:00.603278 9493 catalog_manager.cc:1347] Generated new cluster ID: 76184cb1276644439f42018e740e61a1
I20260621 06:57:00.603415 9493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:00.651098 9493 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:00.651546 9493 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:00.663082 9493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 641dce09473941459a12f6d43c3edde7: Generated new TSK 0
I20260621 06:57:00.664137 9493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:00.682890 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9407
W20260621 06:57:00.684021 9469 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36223
W20260621 06:57:00.684799 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9407: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:57:00.684952 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 9407
2026-06-21T06:57:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:00 dist-test-slave-gmf2 krb5kdc[9505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:00 dist-test-slave-gmf2 krb5kdc[9505](info): set up 2 sockets
Jun 21 06:57:00 dist-test-slave-gmf2 krb5kdc[9505](info): commencing operation
krb5kdc: starting...
W20260621 06:57:01.810266 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:01 dist-test-slave-gmf2 krb5kdc[9505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 06:57:02.049196 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:02.385850 9518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:02.386199 9518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:02.386292 9518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:02.393359 9518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:02.393461 9518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:02.393507 9518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:02.393567 9518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:02.393610 9518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:02.405571 9518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:02.406821 9518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:02.408075 9518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:02.419351 9523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:02.419420 9524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:02.419415 9526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:02.420418 9518 server_base.cc:1034] running on GCE node
I20260621 06:57:02.421204 9518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:02.422646 9518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:02.423893 9518 hybrid_clock.cc:648] HybridClock initialized: now 1782025022423825 us; error 55 us; skew 500 ppm
I20260621 06:57:02.424252 9518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:02 dist-test-slave-gmf2 krb5kdc[9505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:02.427431 9518 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 06:57:02.428938 9518 webserver.cc:460] Webserver started at http://127.0.0.1:35905/ using document root <none> and password file <none>
I20260621 06:57:02.429637 9518 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:02.429780 9518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:02.430154 9518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:02.433066 9518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "e2d2543e70ad47e8a93dffb26f56ec7a"
format_stamp: "Formatted at 2026-06-21 06:57:02 on dist-test-slave-gmf2"
server_key: "f38cc72d599f002c5e481e1253c4a707"
server_key_iv: "dcf72d751332b14b1ced7fea9881d9c3"
server_key_version: "encryptionkey@0"
I20260621 06:57:02.433822 9518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "e2d2543e70ad47e8a93dffb26f56ec7a"
format_stamp: "Formatted at 2026-06-21 06:57:02 on dist-test-slave-gmf2"
server_key: "f38cc72d599f002c5e481e1253c4a707"
server_key_iv: "dcf72d751332b14b1ced7fea9881d9c3"
server_key_version: "encryptionkey@0"
I20260621 06:57:02.438760 9518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 06:57:02.441876 9533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:02.443182 9518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:57:02.443359 9518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "e2d2543e70ad47e8a93dffb26f56ec7a"
format_stamp: "Formatted at 2026-06-21 06:57:02 on dist-test-slave-gmf2"
server_key: "f38cc72d599f002c5e481e1253c4a707"
server_key_iv: "dcf72d751332b14b1ced7fea9881d9c3"
server_key_version: "encryptionkey@0"
I20260621 06:57:02.443539 9518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:02.459985 9518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:02.463670 9518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:02.463938 9518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:02.481802 9518 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38023
I20260621 06:57:02.481833 9584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38023 every 8 connection(s)
I20260621 06:57:02.483126 9518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:02.486650 9585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:02.492720 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 9518
I20260621 06:57:02.492908 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:02.493325 20963 external_mini_cluster.cc:1428] Setting key d9a6ed0773b52a067462343879ee8d2d
I20260621 06:57:02.497689 9585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a: Bootstrap starting.
I20260621 06:57:02.501345 9585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:02.502562 9585 log.cc:824] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:02.505688 9585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a: No bootstrap required, opened a new log
Jun 21 06:57:02 dist-test-slave-gmf2 krb5kdc[9505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 06:57:02.514004 9585 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d2543e70ad47e8a93dffb26f56ec7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38023 } }
I20260621 06:57:02.514468 9585 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:02.514587 9585 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d2543e70ad47e8a93dffb26f56ec7a, State: Initialized, Role: FOLLOWER
I20260621 06:57:02.515203 9585 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d2543e70ad47e8a93dffb26f56ec7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38023 } }
I20260621 06:57:02.515473 9585 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:02.515642 9585 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:02.515830 9585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:02.517251 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:02.495177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:33199 (local address 127.0.0.1:38023)
0621 06:57:02.495901 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:02.495915 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:02.495959 (+ 44us) server_negotiation.cc:407] Connection header received
0621 06:57:02.498363 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:02.498388 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:02.498830 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:02.499258 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:02.500405 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:02.502437 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:02.504109 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:02.505111 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:02.505283 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:02.509663 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:02.509671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:02.509684 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:02.509715 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:02.512832 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:02.513691 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:02.513697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:02.513702 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:02.513851 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:02.514342 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:02.514346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:02.514347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:02.515000 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:02.515253 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:02.515825 (+ 572us) server_negotiation.cc:299] Negotiation successful
0621 06:57:02.516169 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":291,"threads_started":1}
I20260621 06:57:02.518393 9585 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d2543e70ad47e8a93dffb26f56ec7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38023 } }
I20260621 06:57:02.518853 9585 leader_election.cc:302] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2d2543e70ad47e8a93dffb26f56ec7a; no voters:
I20260621 06:57:02.519563 9585 leader_election.cc:288] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:02.519660 9590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:02.520862 9590 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [term 1 LEADER]: Becoming Leader. State: Replica: e2d2543e70ad47e8a93dffb26f56ec7a, State: Running, Role: LEADER
I20260621 06:57:02.521013 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:02.521409 9590 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d2543e70ad47e8a93dffb26f56ec7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38023 } }
I20260621 06:57:02.521816 9585 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:02.524279 9591 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2d2543e70ad47e8a93dffb26f56ec7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d2543e70ad47e8a93dffb26f56ec7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38023 } } }
I20260621 06:57:02.524682 9591 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:02.528676 9592 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2d2543e70ad47e8a93dffb26f56ec7a. Latest consensus state: current_term: 1 leader_uuid: "e2d2543e70ad47e8a93dffb26f56ec7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d2543e70ad47e8a93dffb26f56ec7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38023 } } }
I20260621 06:57:02.529014 9592 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:02.531505 9596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:02.537719 9596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:02.552623 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:02.532702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:50865 (local address 127.0.0.1:38023)
0621 06:57:02.532856 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:02.532863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:02.533570 (+ 707us) server_negotiation.cc:407] Connection header received
0621 06:57:02.533880 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:02.533889 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:02.534062 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:02.534147 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:02.536273 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:02.537440 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:02.539276 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:02.539926 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:02.540021 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:02.543503 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:02.543510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:02.543515 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:02.543551 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:02.547829 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:02.549204 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:02.549212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:02.549218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:02.549404 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:02.550153 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:02.550161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:02.550166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:02.550649 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:02.550934 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:02.551797 (+ 863us) server_negotiation.cc:299] Negotiation successful
0621 06:57:02.552058 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 06:57:02.555790 9596 catalog_manager.cc:1347] Generated new cluster ID: 690d0dbe603842e780ad136b2586dbaa
I20260621 06:57:02.555913 9596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:02.568835 9596 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:02.570246 9596 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:02.580309 9596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2d2543e70ad47e8a93dffb26f56ec7a: Generated new TSK 0
I20260621 06:57:02.581182 9596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:02.588063 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9518
I20260621 06:57:02.589779 9580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:57:02.679044 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 9518
W20260621 06:57:02.696185 9503 connection.cc:535] client connection to 127.0.0.1:38023 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38023): Connection reset by peer (error 104)
2026-06-21T06:57:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:02 dist-test-slave-gmf2 krb5kdc[9617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:02 dist-test-slave-gmf2 krb5kdc[9617](info): set up 2 sockets
Jun 21 06:57:02 dist-test-slave-gmf2 krb5kdc[9617](info): commencing operation
krb5kdc: starting...
W20260621 06:57:03.794433 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:03 dist-test-slave-gmf2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 06:57:04.025197 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36649
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:04.343076 9630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:04.343385 9630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:04.343585 9630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:04.350893 9630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:04.351032 9630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:04.351128 9630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:04.351243 9630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:04.351333 9630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:04.361280 9630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:04.362673 9630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:04.364105 9630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:04.376369 9635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:04.376344 9636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:04.376408 9638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:04.377450 9630 server_base.cc:1034] running on GCE node
I20260621 06:57:04.378306 9630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:04.379675 9630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:04.380889 9630 hybrid_clock.cc:648] HybridClock initialized: now 1782025024380852 us; error 47 us; skew 500 ppm
I20260621 06:57:04.381253 9630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:04 dist-test-slave-gmf2 krb5kdc[9617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:04.384236 9630 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 06:57:04.385522 9630 webserver.cc:460] Webserver started at http://127.0.0.1:33399/ using document root <none> and password file <none>
I20260621 06:57:04.386248 9630 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:04.386346 9630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:04.386586 9630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:04.389307 9630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "cc2be2121c0844bd968369cb2f16a62f"
format_stamp: "Formatted at 2026-06-21 06:57:04 on dist-test-slave-gmf2"
server_key: "46bbecc9335e79356bd9a2ac92eb103c"
server_key_iv: "ce0f6c4c2215e15be1f9cda0bd06c463"
server_key_version: "encryptionkey@0"
I20260621 06:57:04.390065 9630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "cc2be2121c0844bd968369cb2f16a62f"
format_stamp: "Formatted at 2026-06-21 06:57:04 on dist-test-slave-gmf2"
server_key: "46bbecc9335e79356bd9a2ac92eb103c"
server_key_iv: "ce0f6c4c2215e15be1f9cda0bd06c463"
server_key_version: "encryptionkey@0"
I20260621 06:57:04.394642 9630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 06:57:04.397735 9645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:04.398910 9630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 06:57:04.399053 9630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "cc2be2121c0844bd968369cb2f16a62f"
format_stamp: "Formatted at 2026-06-21 06:57:04 on dist-test-slave-gmf2"
server_key: "46bbecc9335e79356bd9a2ac92eb103c"
server_key_iv: "ce0f6c4c2215e15be1f9cda0bd06c463"
server_key_version: "encryptionkey@0"
I20260621 06:57:04.399236 9630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:04.419322 9630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:04.423053 9630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:04.423272 9630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:04.440649 9630 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36649
I20260621 06:57:04.440682 9696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36649 every 8 connection(s)
I20260621 06:57:04.441855 9630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:04.445317 9697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:04.446921 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 9630
I20260621 06:57:04.447110 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:04.447546 20963 external_mini_cluster.cc:1428] Setting key 6c91c6e31974531f41f38886b8c13a16
I20260621 06:57:04.456648 9697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f: Bootstrap starting.
I20260621 06:57:04.459105 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:04.449177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:33749 (local address 127.0.0.1:36649)
0621 06:57:04.449865 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:04.449878 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:04.449956 (+ 78us) server_negotiation.cc:407] Connection header received
0621 06:57:04.451370 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:04.451392 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:04.451770 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:04.452097 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:04.452575 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:04.453827 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:04.455105 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:04.455702 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:04.455887 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:04.457074 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:04.457081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:04.457093 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:04.457581 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:04.457648 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:04.457673 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:04.457842 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:04.458121 (+ 279us) server_negotiation.cc:299] Negotiation successful
0621 06:57:04.458272 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":344,"threads_started":1}
I20260621 06:57:04.459906 9697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:04.461045 9697 log.cc:824] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:04.462852 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:04.463714 9697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f: No bootstrap required, opened a new log
I20260621 06:57:04.473115 9697 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2be2121c0844bd968369cb2f16a62f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36649 } }
I20260621 06:57:04.473587 9697 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:04.473703 9697 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2be2121c0844bd968369cb2f16a62f, State: Initialized, Role: FOLLOWER
I20260621 06:57:04.474421 9697 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2be2121c0844bd968369cb2f16a62f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36649 } }
I20260621 06:57:04.474732 9697 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:04.474879 9697 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:04.475112 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:04.468215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:58569 (local address 127.0.0.1:36649)
0621 06:57:04.468429 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:04.468437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:04.468974 (+ 537us) server_negotiation.cc:407] Connection header received
0621 06:57:04.469149 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:04.469152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:04.469211 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:04.469295 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:04.469853 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:04.470620 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:04.472411 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:04.472829 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:04.472914 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:04.474213 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:04.474228 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:04.474235 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:04.474254 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:04.474339 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:04.474358 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:04.474530 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:04.474678 (+ 148us) server_negotiation.cc:299] Negotiation successful
0621 06:57:04.474736 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 06:57:04.475085 9697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:04.477355 9697 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2be2121c0844bd968369cb2f16a62f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36649 } }
I20260621 06:57:04.477777 9697 leader_election.cc:302] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc2be2121c0844bd968369cb2f16a62f; no voters:
I20260621 06:57:04.478374 9697 leader_election.cc:288] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:04.478528 9707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:04.479291 9707 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [term 1 LEADER]: Becoming Leader. State: Replica: cc2be2121c0844bd968369cb2f16a62f, State: Running, Role: LEADER
I20260621 06:57:04.479940 9707 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2be2121c0844bd968369cb2f16a62f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36649 } }
I20260621 06:57:04.481537 9697 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:04.483197 9708 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc2be2121c0844bd968369cb2f16a62f. Latest consensus state: current_term: 1 leader_uuid: "cc2be2121c0844bd968369cb2f16a62f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2be2121c0844bd968369cb2f16a62f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36649 } } }
I20260621 06:57:04.483654 9708 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:04.484047 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc2be2121c0844bd968369cb2f16a62f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2be2121c0844bd968369cb2f16a62f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36649 } } }
I20260621 06:57:04.484440 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:04.485529 9712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:04.490697 9712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:04.499569 9712 catalog_manager.cc:1347] Generated new cluster ID: 670158720276486497d200e741529370
I20260621 06:57:04.499670 9712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:04.512451 9712 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:04.513609 9712 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:04.520215 9712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc2be2121c0844bd968369cb2f16a62f: Generated new TSK 0
I20260621 06:57:04.520949 9712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:04.527756 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9630
W20260621 06:57:04.528820 9692 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.185:33749
W20260621 06:57:04.529532 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9630: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:57:04.529673 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 9630
W20260621 06:57:04.545837 9615 connection.cc:535] client connection to 127.0.0.1:36649 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36649): Connection reset by peer (error 104)
2026-06-21T06:57:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1849 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:04 dist-test-slave-gmf2 krb5kdc[9728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:04 dist-test-slave-gmf2 krb5kdc[9728](info): set up 2 sockets
Jun 21 06:57:04 dist-test-slave-gmf2 krb5kdc[9728](info): commencing operation
krb5kdc: starting...
W20260621 06:57:05.622411 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:05 dist-test-slave-gmf2 krb5kdc[9728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 06:57:05.870579 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:06.194795 9741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:06.195118 9741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:06.195277 9741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:06.202375 9741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:06.202541 9741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:06.202637 9741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:06.202742 9741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:06.202826 9741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:06.212188 9741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:06.213639 9741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:06.214956 9741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:06.226671 9746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:06.226706 9747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:06.226630 9749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:06.227340 9741 server_base.cc:1034] running on GCE node
I20260621 06:57:06.228091 9741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:06.229449 9741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:06.230661 9741 hybrid_clock.cc:648] HybridClock initialized: now 1782025026230601 us; error 56 us; skew 500 ppm
I20260621 06:57:06.231009 9741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:06 dist-test-slave-gmf2 krb5kdc[9728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:06.238093 9741 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 06:57:06.239524 9741 webserver.cc:460] Webserver started at http://127.0.0.1:33891/ using document root <none> and password file <none>
I20260621 06:57:06.240207 9741 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:06.240305 9741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:06.240564 9741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:06.243635 9741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "80bd132dbcb74f188b24d96d7dec1921"
format_stamp: "Formatted at 2026-06-21 06:57:06 on dist-test-slave-gmf2"
server_key: "3b7e6ece0efdd0e59c421884d89e4f16"
server_key_iv: "55826d412d5db87877265142148c8550"
server_key_version: "encryptionkey@0"
I20260621 06:57:06.244335 9741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "80bd132dbcb74f188b24d96d7dec1921"
format_stamp: "Formatted at 2026-06-21 06:57:06 on dist-test-slave-gmf2"
server_key: "3b7e6ece0efdd0e59c421884d89e4f16"
server_key_iv: "55826d412d5db87877265142148c8550"
server_key_version: "encryptionkey@0"
I20260621 06:57:06.249240 9741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 06:57:06.252601 9756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:06.253949 9741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 06:57:06.254119 9741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "80bd132dbcb74f188b24d96d7dec1921"
format_stamp: "Formatted at 2026-06-21 06:57:06 on dist-test-slave-gmf2"
server_key: "3b7e6ece0efdd0e59c421884d89e4f16"
server_key_iv: "55826d412d5db87877265142148c8550"
server_key_version: "encryptionkey@0"
I20260621 06:57:06.254281 9741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:06.259881 9741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:06.264068 9741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:06.264315 9741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:06.281690 9741 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43071
I20260621 06:57:06.281733 9807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43071 every 8 connection(s)
I20260621 06:57:06.283074 9741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:06.284389 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 9741
I20260621 06:57:06.284555 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:06.284943 20963 external_mini_cluster.cc:1428] Setting key 115444e424d7facfb66832aef2b4653c
I20260621 06:57:06.286974 9808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:06.292366 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:06.286548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:53671 (local address 127.0.0.1:43071)
0621 06:57:06.287354 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:06.287365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:06.287404 (+ 39us) server_negotiation.cc:407] Connection header received
0621 06:57:06.288976 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:06.288997 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:06.289362 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:06.289687 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:06.290834 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:06.290841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:06.290856 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:06.290869 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:06.290940 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:06.290962 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:06.291120 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:06.291232 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 06:57:06.291428 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":355,"threads_started":1}
I20260621 06:57:06.295804 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:06.299430 9808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921: Bootstrap starting.
I20260621 06:57:06.303336 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:06.300945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:35441 (local address 127.0.0.1:43071)
0621 06:57:06.301189 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:06.301194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:06.301532 (+ 338us) server_negotiation.cc:407] Connection header received
0621 06:57:06.301694 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:06.301697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:06.301746 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:06.301820 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:06.302692 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:06.302695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:06.302698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:06.302712 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:06.302745 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:06.302777 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:06.302861 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:06.302999 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 06:57:06.303044 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260621 06:57:06.303409 9808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:06.304477 9808 log.cc:824] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:06.306944 9808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921: No bootstrap required, opened a new log
I20260621 06:57:06.313637 9808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bd132dbcb74f188b24d96d7dec1921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43071 } }
I20260621 06:57:06.314035 9808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:06.314158 9808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80bd132dbcb74f188b24d96d7dec1921, State: Initialized, Role: FOLLOWER
I20260621 06:57:06.314688 9808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bd132dbcb74f188b24d96d7dec1921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43071 } }
I20260621 06:57:06.314904 9808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:06.315066 9808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:06.315235 9808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:06.317339 9808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bd132dbcb74f188b24d96d7dec1921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43071 } }
I20260621 06:57:06.318137 9808 leader_election.cc:302] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80bd132dbcb74f188b24d96d7dec1921; no voters:
I20260621 06:57:06.318768 9808 leader_election.cc:288] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:06.318826 9818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:06.319729 9818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [term 1 LEADER]: Becoming Leader. State: Replica: 80bd132dbcb74f188b24d96d7dec1921, State: Running, Role: LEADER
I20260621 06:57:06.320292 9818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bd132dbcb74f188b24d96d7dec1921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43071 } }
I20260621 06:57:06.321331 9808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:06.323180 9820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80bd132dbcb74f188b24d96d7dec1921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bd132dbcb74f188b24d96d7dec1921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43071 } } }
I20260621 06:57:06.323551 9820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:06.323256 9819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80bd132dbcb74f188b24d96d7dec1921. Latest consensus state: current_term: 1 leader_uuid: "80bd132dbcb74f188b24d96d7dec1921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80bd132dbcb74f188b24d96d7dec1921" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43071 } } }
I20260621 06:57:06.323944 9819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:06.331768 9823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:06.338255 9823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:06.347456 9823 catalog_manager.cc:1347] Generated new cluster ID: 3e22f70877444172a7c85b6e05225d5f
I20260621 06:57:06.347579 9823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:06.380617 9823 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:06.381129 9823 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:06.389439 9823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80bd132dbcb74f188b24d96d7dec1921: Generated new TSK 0
I20260621 06:57:06.390297 9823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:06.397294 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9741
W20260621 06:57:06.398243 9803 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.185:53671
W20260621 06:57:06.398885 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9741: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:57:06.399012 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 9741
2026-06-21T06:57:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:06 dist-test-slave-gmf2 krb5kdc[9839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:06 dist-test-slave-gmf2 krb5kdc[9839](info): set up 2 sockets
Jun 21 06:57:06 dist-test-slave-gmf2 krb5kdc[9839](info): commencing operation
krb5kdc: starting...
W20260621 06:57:07.474273 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:07 dist-test-slave-gmf2 krb5kdc[9839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 06:57:07.710924 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:08.032929 9852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:08.033279 9852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:08.033452 9852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:08.040280 9852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:08.040421 9852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:08.040516 9852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:08.040632 9852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:08.040709 9852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:08.049935 9852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:08.051227 9852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:08.052573 9852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:08.065585 9857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:08.065627 9858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:08.065672 9860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:08.066473 9852 server_base.cc:1034] running on GCE node
I20260621 06:57:08.067137 9852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:08.068382 9852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:08.069573 9852 hybrid_clock.cc:648] HybridClock initialized: now 1782025028069517 us; error 45 us; skew 500 ppm
I20260621 06:57:08.070108 9852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:08 dist-test-slave-gmf2 krb5kdc[9839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:08.073331 9852 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 06:57:08.074753 9852 webserver.cc:460] Webserver started at http://127.0.0.1:39387/ using document root <none> and password file <none>
I20260621 06:57:08.075383 9852 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:08.075524 9852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:08.075824 9852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:08.078763 9852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "01abd2939d584b44914b35378ca1bba8"
format_stamp: "Formatted at 2026-06-21 06:57:08 on dist-test-slave-gmf2"
server_key: "3bdcbbb360facd99cd44e5600dd474fe"
server_key_iv: "cc55bf7cbb21eeaf6618229fa9ee4d5b"
server_key_version: "encryptionkey@0"
I20260621 06:57:08.079586 9852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "01abd2939d584b44914b35378ca1bba8"
format_stamp: "Formatted at 2026-06-21 06:57:08 on dist-test-slave-gmf2"
server_key: "3bdcbbb360facd99cd44e5600dd474fe"
server_key_iv: "cc55bf7cbb21eeaf6618229fa9ee4d5b"
server_key_version: "encryptionkey@0"
I20260621 06:57:08.084635 9852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 06:57:08.087899 9867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:08.088984 9852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 06:57:08.089188 9852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "01abd2939d584b44914b35378ca1bba8"
format_stamp: "Formatted at 2026-06-21 06:57:08 on dist-test-slave-gmf2"
server_key: "3bdcbbb360facd99cd44e5600dd474fe"
server_key_iv: "cc55bf7cbb21eeaf6618229fa9ee4d5b"
server_key_version: "encryptionkey@0"
I20260621 06:57:08.089390 9852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:08.117681 9852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:08.122177 9852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:08.122483 9852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:08.141384 9852 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42663
I20260621 06:57:08.141403 9918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42663 every 8 connection(s)
I20260621 06:57:08.142946 9852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:08.146623 9919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:08.153126 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 9852
I20260621 06:57:08.153304 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:08.153698 20963 external_mini_cluster.cc:1428] Setting key 11f691994ad0e7b3e76ecf4a27fe5ed4
I20260621 06:57:08.156503 9919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8: Bootstrap starting.
I20260621 06:57:08.159889 9919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:08.161057 9919 log.cc:824] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:08.163971 9919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8: No bootstrap required, opened a new log
Jun 21 06:57:08 dist-test-slave-gmf2 krb5kdc[9839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 06:57:08.172905 9919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abd2939d584b44914b35378ca1bba8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42663 } }
I20260621 06:57:08.173327 9919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:08.173444 9919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01abd2939d584b44914b35378ca1bba8, State: Initialized, Role: FOLLOWER
I20260621 06:57:08.174099 9919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abd2939d584b44914b35378ca1bba8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42663 } }
I20260621 06:57:08.174396 9919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:08.174553 9919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:08.174731 9919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:08.176283 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:08.155346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:60875 (local address 127.0.0.1:42663)
0621 06:57:08.156112 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:08.156123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:08.156165 (+ 42us) server_negotiation.cc:407] Connection header received
0621 06:57:08.157834 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:08.157858 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:08.158334 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:08.158660 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:08.160537 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:08.161881 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:08.163251 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:08.164247 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:08.164439 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:08.168191 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:08.168200 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:08.168217 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:08.168272 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:08.172113 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:08.172784 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:08.172791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:08.172798 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:08.173033 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:08.173511 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:08.173517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:08.173520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:08.174192 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:08.174504 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:08.175024 (+ 520us) server_negotiation.cc:299] Negotiation successful
0621 06:57:08.175330 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":364,"threads_started":1}
I20260621 06:57:08.177472 9919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abd2939d584b44914b35378ca1bba8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42663 } }
I20260621 06:57:08.177989 9919 leader_election.cc:302] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01abd2939d584b44914b35378ca1bba8; no voters:
I20260621 06:57:08.178560 9919 leader_election.cc:288] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:08.178691 9924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:08.179987 9924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [term 1 LEADER]: Becoming Leader. State: Replica: 01abd2939d584b44914b35378ca1bba8, State: Running, Role: LEADER
I20260621 06:57:08.180433 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:08.180558 9924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abd2939d584b44914b35378ca1bba8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42663 } }
I20260621 06:57:08.180718 9919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:08.183000 9926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01abd2939d584b44914b35378ca1bba8. Latest consensus state: current_term: 1 leader_uuid: "01abd2939d584b44914b35378ca1bba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abd2939d584b44914b35378ca1bba8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42663 } } }
I20260621 06:57:08.183370 9926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:08.183899 9925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01abd2939d584b44914b35378ca1bba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01abd2939d584b44914b35378ca1bba8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42663 } } }
I20260621 06:57:08.184195 9925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:08.192478 9929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:08.198642 9929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:08.211655 9929 catalog_manager.cc:1347] Generated new cluster ID: 70f38d8798d14cbaa085f96d8ca67871
I20260621 06:57:08.211788 9929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:08.229310 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:08.209367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:46477 (local address 127.0.0.1:42663)
0621 06:57:08.209549 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:08.209555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:08.213155 (+ 3600us) server_negotiation.cc:407] Connection header received
0621 06:57:08.213343 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:08.213350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:08.213439 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:08.213578 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:08.215405 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:08.216523 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:08.218232 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:08.218815 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:08.218894 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:08.222188 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:08.222196 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:08.222200 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:08.222235 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:08.226171 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:08.226995 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:08.227000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:08.227004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:08.227171 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:08.227651 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:08.227657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:08.227660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:08.227974 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:08.228198 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:08.228805 (+ 607us) server_negotiation.cc:299] Negotiation successful
0621 06:57:08.228978 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 06:57:08.239638 9929 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:08.240906 9929 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:08.255827 9929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01abd2939d584b44914b35378ca1bba8: Generated new TSK 0
I20260621 06:57:08.256603 9929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:08.263012 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9852
I20260621 06:57:08.264635 9914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:57:08.353946 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 9852
W20260621 06:57:08.371729 9837 connection.cc:535] client connection to 127.0.0.1:42663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42663): Connection reset by peer (error 104)
2026-06-21T06:57:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:08 dist-test-slave-gmf2 krb5kdc[9952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:08 dist-test-slave-gmf2 krb5kdc[9952](info): set up 2 sockets
Jun 21 06:57:08 dist-test-slave-gmf2 krb5kdc[9952](info): commencing operation
krb5kdc: starting...
W20260621 06:57:09.429824 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:09 dist-test-slave-gmf2 krb5kdc[9952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 06:57:09.658658 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:09.971876 9965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:09.972119 9965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:09.972208 9965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:09.978754 9965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:09.978844 9965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:09.978886 9965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:09.978941 9965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:09.978982 9965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:09.988021 9965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:09.989125 9965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:09.990391 9965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:10.001972 9973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:10.002027 9970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:10.002007 9971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:10.003084 9965 server_base.cc:1034] running on GCE node
I20260621 06:57:10.003885 9965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:10.005235 9965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:10.006429 9965 hybrid_clock.cc:648] HybridClock initialized: now 1782025030006385 us; error 42 us; skew 500 ppm
I20260621 06:57:10.006767 9965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:10 dist-test-slave-gmf2 krb5kdc[9952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:10.009816 9965 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 06:57:10.011323 9965 webserver.cc:460] Webserver started at http://127.0.0.1:44839/ using document root <none> and password file <none>
I20260621 06:57:10.011989 9965 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:10.012122 9965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:10.012424 9965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:10.015512 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "9ab035de3d8f404abfb930a69ff4c079"
format_stamp: "Formatted at 2026-06-21 06:57:10 on dist-test-slave-gmf2"
server_key: "9ee9d4edfbff3acc47231972671372ab"
server_key_iv: "0a90e9ed96d2f2bf7f5c2b67f8cd2868"
server_key_version: "encryptionkey@0"
I20260621 06:57:10.016261 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "9ab035de3d8f404abfb930a69ff4c079"
format_stamp: "Formatted at 2026-06-21 06:57:10 on dist-test-slave-gmf2"
server_key: "9ee9d4edfbff3acc47231972671372ab"
server_key_iv: "0a90e9ed96d2f2bf7f5c2b67f8cd2868"
server_key_version: "encryptionkey@0"
I20260621 06:57:10.021359 9965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 06:57:10.024693 9980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:10.026026 9965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:57:10.026190 9965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "9ab035de3d8f404abfb930a69ff4c079"
format_stamp: "Formatted at 2026-06-21 06:57:10 on dist-test-slave-gmf2"
server_key: "9ee9d4edfbff3acc47231972671372ab"
server_key_iv: "0a90e9ed96d2f2bf7f5c2b67f8cd2868"
server_key_version: "encryptionkey@0"
I20260621 06:57:10.026371 9965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:10.055661 9965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:10.059857 9965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:10.060159 9965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:10.079365 9965 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46803
I20260621 06:57:10.079386 10031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46803 every 8 connection(s)
I20260621 06:57:10.080821 9965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:10.084491 10032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:10.089479 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 9965
I20260621 06:57:10.089638 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:10.090118 20963 external_mini_cluster.cc:1428] Setting key b4c3fec7d1d510e66d0933584d395881
I20260621 06:57:10.094646 10032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079: Bootstrap starting.
I20260621 06:57:10.098117 10032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:10.099246 10032 log.cc:824] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:10.101889 10032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079: No bootstrap required, opened a new log
I20260621 06:57:10.103099 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:10.091854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:37751 (local address 127.0.0.1:46803)
0621 06:57:10.092620 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:10.092630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:10.092687 (+ 57us) server_negotiation.cc:407] Connection header received
0621 06:57:10.094352 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:10.094373 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:10.094748 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:10.095097 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:10.095656 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:10.096906 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:10.098756 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:10.099466 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:10.099588 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:10.100883 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:10.100897 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:10.100911 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:10.101433 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:10.101500 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:10.101523 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:10.101678 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:10.102003 (+ 325us) server_negotiation.cc:299] Negotiation successful
0621 06:57:10.102159 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":345,"threads_started":1}
I20260621 06:57:10.106916 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:10.110333 10032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab035de3d8f404abfb930a69ff4c079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46803 } }
I20260621 06:57:10.110769 10032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:10.110888 10032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ab035de3d8f404abfb930a69ff4c079, State: Initialized, Role: FOLLOWER
I20260621 06:57:10.111621 10032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab035de3d8f404abfb930a69ff4c079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46803 } }
I20260621 06:57:10.111881 10032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:10.112032 10032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:10.112214 10032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:10.114578 10032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab035de3d8f404abfb930a69ff4c079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46803 } }
I20260621 06:57:10.115095 10032 leader_election.cc:302] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ab035de3d8f404abfb930a69ff4c079; no voters:
I20260621 06:57:10.115749 10032 leader_election.cc:288] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:10.115847 10042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:10.117477 10042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [term 1 LEADER]: Becoming Leader. State: Replica: 9ab035de3d8f404abfb930a69ff4c079, State: Running, Role: LEADER
I20260621 06:57:10.117946 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:10.111118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:48501 (local address 127.0.0.1:46803)
0621 06:57:10.111296 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:10.111304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:10.111738 (+ 434us) server_negotiation.cc:407] Connection header received
0621 06:57:10.111933 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:10.111938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:10.112018 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:10.112125 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:10.112657 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:10.113456 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:10.114790 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:10.115287 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:10.115456 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:10.116645 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:10.116652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:10.116657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:10.116678 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:10.116751 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:10.116772 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:10.117429 (+ 657us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:10.117559 (+ 130us) server_negotiation.cc:299] Negotiation successful
0621 06:57:10.117632 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 06:57:10.118147 10042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab035de3d8f404abfb930a69ff4c079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46803 } }
I20260621 06:57:10.118470 10032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:10.121328 10044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ab035de3d8f404abfb930a69ff4c079. Latest consensus state: current_term: 1 leader_uuid: "9ab035de3d8f404abfb930a69ff4c079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab035de3d8f404abfb930a69ff4c079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46803 } } }
I20260621 06:57:10.121364 10043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ab035de3d8f404abfb930a69ff4c079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab035de3d8f404abfb930a69ff4c079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46803 } } }
I20260621 06:57:10.121768 10044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:10.121779 10043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:10.123581 10050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:10.128798 10050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:10.138468 10050 catalog_manager.cc:1347] Generated new cluster ID: 322e267884c64ae4aff76aa8576e4b12
I20260621 06:57:10.138586 10050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:10.149834 10050 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:10.151063 10050 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:10.160852 10050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ab035de3d8f404abfb930a69ff4c079: Generated new TSK 0
I20260621 06:57:10.161799 10050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:10.175067 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9965
W20260621 06:57:10.176122 10027 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.185:37751
W20260621 06:57:10.176831 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 9965: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:57:10.176983 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 9965
W20260621 06:57:10.193276 9950 connection.cc:535] client connection to 127.0.0.1:46803 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46803): Connection reset by peer (error 104)
2026-06-21T06:57:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1821 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:10 dist-test-slave-gmf2 krb5kdc[10063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:10 dist-test-slave-gmf2 krb5kdc[10063](info): set up 2 sockets
Jun 21 06:57:10 dist-test-slave-gmf2 krb5kdc[10063](info): commencing operation
krb5kdc: starting...
W20260621 06:57:11.314383 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:11 dist-test-slave-gmf2 krb5kdc[10063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 06:57:11.567022 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42841
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:11.887562 10076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:11.887835 10076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:11.887938 10076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:11.894629 10076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:11.894712 10076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:11.894759 10076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:11.894812 10076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:11.894851 10076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:11.903748 10076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42841
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:11.904824 10076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:11.905984 10076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:11.916914 10084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:11.916985 10081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:11.917007 10082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:11.917945 10076 server_base.cc:1034] running on GCE node
I20260621 06:57:11.918576 10076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:11.919741 10076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:11.920979 10076 hybrid_clock.cc:648] HybridClock initialized: now 1782025031920909 us; error 41 us; skew 500 ppm
I20260621 06:57:11.921312 10076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:11 dist-test-slave-gmf2 krb5kdc[10063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:11.928400 10076 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 06:57:11.929589 10076 webserver.cc:460] Webserver started at http://127.0.0.1:43101/ using document root <none> and password file <none>
I20260621 06:57:11.930246 10076 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:11.930358 10076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:11.930579 10076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:11.933281 10076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "0b92122c75b34f828084150d184dc5b4"
format_stamp: "Formatted at 2026-06-21 06:57:11 on dist-test-slave-gmf2"
server_key: "74f1e4e6413063f25eec9c17edefb0dd"
server_key_iv: "3422455b2bc1e68ec33ba567db53968f"
server_key_version: "encryptionkey@0"
I20260621 06:57:11.934026 10076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "0b92122c75b34f828084150d184dc5b4"
format_stamp: "Formatted at 2026-06-21 06:57:11 on dist-test-slave-gmf2"
server_key: "74f1e4e6413063f25eec9c17edefb0dd"
server_key_iv: "3422455b2bc1e68ec33ba567db53968f"
server_key_version: "encryptionkey@0"
I20260621 06:57:11.939507 10076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 06:57:11.942741 10091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:11.943998 10076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:57:11.944164 10076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "0b92122c75b34f828084150d184dc5b4"
format_stamp: "Formatted at 2026-06-21 06:57:11 on dist-test-slave-gmf2"
server_key: "74f1e4e6413063f25eec9c17edefb0dd"
server_key_iv: "3422455b2bc1e68ec33ba567db53968f"
server_key_version: "encryptionkey@0"
I20260621 06:57:11.944345 10076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:11.949795 10076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:11.953394 10076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:11.953660 10076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:11.971508 10076 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42841
I20260621 06:57:11.971604 10142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42841 every 8 connection(s)
I20260621 06:57:11.972837 10076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:11.976341 10143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:11.978567 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 10076
I20260621 06:57:11.978724 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:11.979240 20963 external_mini_cluster.cc:1428] Setting key 5edbcecc6b1a49d874c6b63dc7c59af7
I20260621 06:57:11.986644 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:11.980882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:57983 (local address 127.0.0.1:42841)
0621 06:57:11.981673 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:11.981693 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:11.981735 (+ 42us) server_negotiation.cc:407] Connection header received
0621 06:57:11.983433 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:11.983454 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:11.983770 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:11.984068 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:11.985079 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:11.985084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:11.985100 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:11.985114 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:11.985175 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:11.985198 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:11.985391 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:11.985485 (+ 94us) server_negotiation.cc:299] Negotiation successful
0621 06:57:11.985713 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":373,"threads_started":1}
I20260621 06:57:11.988165 10143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4: Bootstrap starting.
I20260621 06:57:11.990029 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:11.991515 10143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:11.992801 10143 log.cc:824] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:11.995939 10143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4: No bootstrap required, opened a new log
I20260621 06:57:11.998760 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:11.995984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:33943 (local address 127.0.0.1:42841)
0621 06:57:11.996134 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:11.996138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:11.996725 (+ 587us) server_negotiation.cc:407] Connection header received
0621 06:57:11.996973 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:11.996977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:11.997023 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:11.997104 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:11.998106 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:11.998113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:11.998116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:11.998131 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:11.998169 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:11.998199 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:11.998290 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:11.998411 (+ 121us) server_negotiation.cc:299] Negotiation successful
0621 06:57:11.998456 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 06:57:12.004034 10143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92122c75b34f828084150d184dc5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42841 } }
I20260621 06:57:12.004424 10143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:12.004534 10143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b92122c75b34f828084150d184dc5b4, State: Initialized, Role: FOLLOWER
I20260621 06:57:12.005163 10143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92122c75b34f828084150d184dc5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42841 } }
I20260621 06:57:12.005398 10143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:12.005532 10143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:12.005694 10143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:12.007877 10143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92122c75b34f828084150d184dc5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42841 } }
I20260621 06:57:12.008342 10143 leader_election.cc:302] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b92122c75b34f828084150d184dc5b4; no voters:
I20260621 06:57:12.008904 10143 leader_election.cc:288] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:12.009078 10153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:12.009832 10153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [term 1 LEADER]: Becoming Leader. State: Replica: 0b92122c75b34f828084150d184dc5b4, State: Running, Role: LEADER
I20260621 06:57:12.010422 10153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92122c75b34f828084150d184dc5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42841 } }
I20260621 06:57:12.010932 10143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:12.014089 10154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b92122c75b34f828084150d184dc5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92122c75b34f828084150d184dc5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42841 } } }
I20260621 06:57:12.014771 10154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:12.014113 10155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b92122c75b34f828084150d184dc5b4. Latest consensus state: current_term: 1 leader_uuid: "0b92122c75b34f828084150d184dc5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b92122c75b34f828084150d184dc5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42841 } } }
I20260621 06:57:12.015560 10155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:12.016786 10162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:12.021433 10162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:12.029537 10162 catalog_manager.cc:1347] Generated new cluster ID: 254456a0ea5f4e96b5c98c3e40b6df6d
I20260621 06:57:12.029656 10162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:12.049095 10162 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:12.049510 10162 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:12.064157 10162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b92122c75b34f828084150d184dc5b4: Generated new TSK 0
I20260621 06:57:12.064908 10162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:12.083607 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10076
W20260621 06:57:12.084622 10138 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.185:57983
W20260621 06:57:12.085224 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10076: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:57:12.085354 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 10076
2026-06-21T06:57:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1905 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:12 dist-test-slave-gmf2 krb5kdc[10174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:12 dist-test-slave-gmf2 krb5kdc[10174](info): set up 2 sockets
Jun 21 06:57:12 dist-test-slave-gmf2 krb5kdc[10174](info): commencing operation
krb5kdc: starting...
W20260621 06:57:13.182351 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:13 dist-test-slave-gmf2 krb5kdc[10174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:57:13.445470 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:36287
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:41563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:36287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:13.786269 10187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:13.786521 10187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:13.786669 10187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:13.793351 10187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:13.793452 10187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:13.793536 10187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:13.793642 10187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:13.793707 10187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:13.803205 10187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:41563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:36287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:36287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:13.804493 10187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:13.805832 10187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:13.817796 10193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:13.817818 10192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:13.818461 10187 server_base.cc:1034] running on GCE node
W20260621 06:57:13.818526 10195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:13.819273 10187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:13.820604 10187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:13.821843 10187 hybrid_clock.cc:648] HybridClock initialized: now 1782025033821791 us; error 42 us; skew 500 ppm
I20260621 06:57:13.822209 10187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:13 dist-test-slave-gmf2 krb5kdc[10174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025033, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:13.825150 10187 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:57:13.826546 10187 webserver.cc:460] Webserver started at http://127.20.120.254:46461/ using document root <none> and password file <none>
I20260621 06:57:13.827126 10187 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:13.827232 10187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:13.827466 10187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:13.830308 10187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "ebcd437215d3430f8c02e93e5645200f"
format_stamp: "Formatted at 2026-06-21 06:57:13 on dist-test-slave-gmf2"
server_key: "5ad1c8c98fff08cbf9696d0a2ed39cac"
server_key_iv: "d01b6df6858d035f3b7c3a31154ff869"
server_key_version: "encryptionkey@0"
I20260621 06:57:13.831001 10187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "ebcd437215d3430f8c02e93e5645200f"
format_stamp: "Formatted at 2026-06-21 06:57:13 on dist-test-slave-gmf2"
server_key: "5ad1c8c98fff08cbf9696d0a2ed39cac"
server_key_iv: "d01b6df6858d035f3b7c3a31154ff869"
server_key_version: "encryptionkey@0"
I20260621 06:57:13.835718 10187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 06:57:13.838850 10202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:13.840078 10187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:57:13.840260 10187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "ebcd437215d3430f8c02e93e5645200f"
format_stamp: "Formatted at 2026-06-21 06:57:13 on dist-test-slave-gmf2"
server_key: "5ad1c8c98fff08cbf9696d0a2ed39cac"
server_key_iv: "d01b6df6858d035f3b7c3a31154ff869"
server_key_version: "encryptionkey@0"
I20260621 06:57:13.840435 10187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:13.866389 10187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:13.870294 10187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:13.870589 10187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:13.888825 10187 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:36287
I20260621 06:57:13.888849 10253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:36287 every 8 connection(s)
I20260621 06:57:13.890137 10187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:13.891968 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 10187
I20260621 06:57:13.892153 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:13.892633 20963 external_mini_cluster.cc:1428] Setting key 70fbe2e3a5d522e1d343472004f9b686
I20260621 06:57:13.894171 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 06:57:13 dist-test-slave-gmf2 krb5kdc[10174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:57:13.907685 10254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f: Bootstrap starting.
I20260621 06:57:13.910755 10254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:13.911720 10254 log.cc:824] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:13.914526 10254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f: No bootstrap required, opened a new log
I20260621 06:57:13.915308 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:13.894464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40365 (local address 127.20.120.254:36287)
0621 06:57:13.895229 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:13.895241 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:13.895280 (+ 39us) server_negotiation.cc:407] Connection header received
0621 06:57:13.896817 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:13.896836 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:13.897211 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:13.897544 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:13.899346 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:13.900671 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:13.902432 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:13.903091 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:13.903242 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:13.907983 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:13.907991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:13.908003 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:13.908040 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:13.911156 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:13.911937 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:13.911943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:13.911948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:13.912119 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:13.912639 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:13.912642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:13.912644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:13.913120 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:13.913448 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:13.914003 (+ 555us) server_negotiation.cc:299] Negotiation successful
0621 06:57:13.914288 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":356,"threads_started":1}
I20260621 06:57:13.919207 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:13.922737 10254 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcd437215d3430f8c02e93e5645200f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36287 } }
I20260621 06:57:13.923142 10254 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:13.923257 10254 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcd437215d3430f8c02e93e5645200f, State: Initialized, Role: FOLLOWER
I20260621 06:57:13.923957 10254 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcd437215d3430f8c02e93e5645200f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36287 } }
I20260621 06:57:13.924271 10254 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:13.924441 10254 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:13.924619 10254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:13.926771 10254 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcd437215d3430f8c02e93e5645200f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36287 } }
I20260621 06:57:13.927258 10254 leader_election.cc:302] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebcd437215d3430f8c02e93e5645200f; no voters:
I20260621 06:57:13.927889 10254 leader_election.cc:288] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:13.927989 10264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:13.930298 10254 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:13.935894 10264 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [term 1 LEADER]: Becoming Leader. State: Replica: ebcd437215d3430f8c02e93e5645200f, State: Running, Role: LEADER
I20260621 06:57:13.937027 10264 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcd437215d3430f8c02e93e5645200f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36287 } }
I20260621 06:57:13.938441 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:13.923281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53665 (local address 127.20.120.254:36287)
0621 06:57:13.923440 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:13.923449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:13.923955 (+ 506us) server_negotiation.cc:407] Connection header received
0621 06:57:13.924117 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:13.924120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:13.924222 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:13.924347 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:13.925702 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:13.926527 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:13.927838 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:13.928543 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:13.928665 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:13.931273 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:13.931278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:13.931288 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:13.931313 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:13.934499 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:13.935410 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:13.935417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:13.935425 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:13.935583 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:13.936075 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:13.936079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:13.936082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:13.936583 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:13.937116 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:13.937972 (+ 856us) server_negotiation.cc:299] Negotiation successful
0621 06:57:13.938141 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 06:57:13.940227 10265 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebcd437215d3430f8c02e93e5645200f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcd437215d3430f8c02e93e5645200f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36287 } } }
I20260621 06:57:13.940575 10265 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:13.942235 10266 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebcd437215d3430f8c02e93e5645200f. Latest consensus state: current_term: 1 leader_uuid: "ebcd437215d3430f8c02e93e5645200f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcd437215d3430f8c02e93e5645200f" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 36287 } } }
I20260621 06:57:13.943360 10266 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:13.946024 10277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:13.949697 10277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:13.957293 10277 catalog_manager.cc:1347] Generated new cluster ID: ca0f85ce5d354a6f81ecf44b68bda723
I20260621 06:57:13.957397 10277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:13.971834 10277 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:13.973490 10277 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:13.979943 10277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebcd437215d3430f8c02e93e5645200f: Generated new TSK 0
I20260621 06:57:13.980665 10277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:13.991880 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10187
I20260621 06:57:13.993221 10249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:57:14.082521 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 10187
W20260621 06:57:14.099292 10172 connection.cc:535] client connection to 127.20.120.254:36287 recv error: Network error: failed to read from TLS socket (remote: 127.20.120.254:36287): Connection reset by peer (error 104)
2026-06-21T06:57:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:14 dist-test-slave-gmf2 krb5kdc[10287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:14 dist-test-slave-gmf2 krb5kdc[10287](info): set up 2 sockets
Jun 21 06:57:14 dist-test-slave-gmf2 krb5kdc[10287](info): commencing operation
krb5kdc: starting...
W20260621 06:57:15.158372 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:15 dist-test-slave-gmf2 krb5kdc[10287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:57:15.382575 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:42187
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:42187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:15.703747 10300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:15.704013 10300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:15.704208 10300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:15.711314 10300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:15.711417 10300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:15.711544 10300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:15.711644 10300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:15.711728 10300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:15.721380 10300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:42187
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:42187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:15.722800 10300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:15.724186 10300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:15.735755 10306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:15.735754 10305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:15.735710 10308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:15.736588 10300 server_base.cc:1034] running on GCE node
I20260621 06:57:15.737247 10300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:15.738569 10300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:15.739768 10300 hybrid_clock.cc:648] HybridClock initialized: now 1782025035739728 us; error 28 us; skew 500 ppm
I20260621 06:57:15.740103 10300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:15 dist-test-slave-gmf2 krb5kdc[10287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025035, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:15.743187 10300 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:57:15.744597 10300 webserver.cc:460] Webserver started at http://127.20.120.254:34745/ using document root <none> and password file <none>
I20260621 06:57:15.745234 10300 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:15.745332 10300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:15.745572 10300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:15.748569 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "6cd2174232db47faa62fbff296d5c1d0"
format_stamp: "Formatted at 2026-06-21 06:57:15 on dist-test-slave-gmf2"
server_key: "764b0034e05b599904be53c49fc070c4"
server_key_iv: "7940cde97ddab706ffcd4e681234d512"
server_key_version: "encryptionkey@0"
I20260621 06:57:15.749250 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "6cd2174232db47faa62fbff296d5c1d0"
format_stamp: "Formatted at 2026-06-21 06:57:15 on dist-test-slave-gmf2"
server_key: "764b0034e05b599904be53c49fc070c4"
server_key_iv: "7940cde97ddab706ffcd4e681234d512"
server_key_version: "encryptionkey@0"
I20260621 06:57:15.754467 10300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 06:57:15.758391 10315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:15.759719 10300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 06:57:15.759896 10300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "6cd2174232db47faa62fbff296d5c1d0"
format_stamp: "Formatted at 2026-06-21 06:57:15 on dist-test-slave-gmf2"
server_key: "764b0034e05b599904be53c49fc070c4"
server_key_iv: "7940cde97ddab706ffcd4e681234d512"
server_key_version: "encryptionkey@0"
I20260621 06:57:15.760092 10300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:15.774022 10300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:15.778057 10300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:15.778352 10300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:15.796386 10300 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:42187
I20260621 06:57:15.796408 10366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:42187 every 8 connection(s)
I20260621 06:57:15.797816 10300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:15.801522 10367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:15.803676 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 10300
I20260621 06:57:15.803870 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:15.804418 20963 external_mini_cluster.cc:1428] Setting key 5c612a1eca7173b32e9479eeb5ea5aee
I20260621 06:57:15.816063 10367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0: Bootstrap starting.
I20260621 06:57:15.818354 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:15.806263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44803 (local address 127.20.120.254:42187)
0621 06:57:15.807033 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:15.807044 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:15.807090 (+ 46us) server_negotiation.cc:407] Connection header received
0621 06:57:15.808614 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:15.808639 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:15.809019 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:15.809336 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:15.809983 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:15.811809 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:15.813644 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:15.814392 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:15.814516 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:15.815802 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:15.815826 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:15.815839 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:15.816415 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:15.816502 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:15.816529 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:15.816638 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:15.817027 (+ 389us) server_negotiation.cc:299] Negotiation successful
0621 06:57:15.817262 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":369,"threads_started":1}
I20260621 06:57:15.819507 10367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:15.820840 10367 log.cc:824] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:15.822376 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:15.823697 10367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0: No bootstrap required, opened a new log
I20260621 06:57:15.832705 10367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cd2174232db47faa62fbff296d5c1d0" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42187 } }
I20260621 06:57:15.833240 10367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:15.833364 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:15.826893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60807 (local address 127.20.120.254:42187)
0621 06:57:15.827079 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:15.827085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:15.827420 (+ 335us) server_negotiation.cc:407] Connection header received
0621 06:57:15.827605 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:15.827612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:15.827704 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:15.827818 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:15.828263 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:15.829436 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:15.830705 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:15.831405 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:15.831525 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:15.832587 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:15.832594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:15.832598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:15.832619 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:15.832677 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:15.832706 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:15.832822 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:15.832983 (+ 161us) server_negotiation.cc:299] Negotiation successful
0621 06:57:15.833034 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:57:15.833369 10367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cd2174232db47faa62fbff296d5c1d0, State: Initialized, Role: FOLLOWER
I20260621 06:57:15.834143 10367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cd2174232db47faa62fbff296d5c1d0" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42187 } }
I20260621 06:57:15.834398 10367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:15.834556 10367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:15.834738 10367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:15.836947 10367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cd2174232db47faa62fbff296d5c1d0" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42187 } }
I20260621 06:57:15.837450 10367 leader_election.cc:302] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cd2174232db47faa62fbff296d5c1d0; no voters:
I20260621 06:57:15.838090 10367 leader_election.cc:288] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:15.838229 10377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:15.838900 10377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [term 1 LEADER]: Becoming Leader. State: Replica: 6cd2174232db47faa62fbff296d5c1d0, State: Running, Role: LEADER
I20260621 06:57:15.839537 10377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cd2174232db47faa62fbff296d5c1d0" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42187 } }
I20260621 06:57:15.840441 10367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:15.842777 10379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cd2174232db47faa62fbff296d5c1d0. Latest consensus state: current_term: 1 leader_uuid: "6cd2174232db47faa62fbff296d5c1d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cd2174232db47faa62fbff296d5c1d0" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42187 } } }
I20260621 06:57:15.843228 10379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:15.845321 10378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cd2174232db47faa62fbff296d5c1d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cd2174232db47faa62fbff296d5c1d0" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 42187 } } }
I20260621 06:57:15.845729 10378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:15.846508 10384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:15.851749 10384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:15.861020 10384 catalog_manager.cc:1347] Generated new cluster ID: bf87b84ae7c44abc9e88e14a55e9cd32
I20260621 06:57:15.861155 10384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:15.874840 10384 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:15.876220 10384 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:15.884440 10384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cd2174232db47faa62fbff296d5c1d0: Generated new TSK 0
I20260621 06:57:15.885352 10384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:15.904031 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10300
W20260621 06:57:15.905229 10362 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44803
W20260621 06:57:15.906098 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10300: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:57:15.906276 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 10300
W20260621 06:57:15.923579 10285 connection.cc:535] client connection to 127.20.120.254:42187 recv error: Network error: failed to read from TLS socket (remote: 127.20.120.254:42187): Connection reset by peer (error 104)
2026-06-21T06:57:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1824 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:16 dist-test-slave-gmf2 krb5kdc[10398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:16 dist-test-slave-gmf2 krb5kdc[10398](info): set up 2 sockets
Jun 21 06:57:16 dist-test-slave-gmf2 krb5kdc[10398](info): commencing operation
krb5kdc: starting...
W20260621 06:57:16.994338 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:17 dist-test-slave-gmf2 krb5kdc[10398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:57:17.247069 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:35887
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:34249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:35887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:17.569844 10411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:17.570206 10411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:17.570371 10411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:17.577518 10411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:17.577684 10411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:17.577796 10411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:17.577901 10411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:17.577999 10411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:17.587949 10411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:34249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:35887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:17.589362 10411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:17.590811 10411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:17.602968 10416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:17.602942 10417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:17.603477 10419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:17.603860 10411 server_base.cc:1034] running on GCE node
I20260621 06:57:17.604633 10411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:17.606232 10411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:17.607432 10411 hybrid_clock.cc:648] HybridClock initialized: now 1782025037607395 us; error 35 us; skew 500 ppm
I20260621 06:57:17.607815 10411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:17 dist-test-slave-gmf2 krb5kdc[10398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025037, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:17.613754 10411 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:57:17.615387 10411 webserver.cc:460] Webserver started at http://127.20.120.254:45519/ using document root <none> and password file <none>
I20260621 06:57:17.616073 10411 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:17.616226 10411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:17.616554 10411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:17.619625 10411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "5fed1a26e56c4c2ea5df22756070b42e"
format_stamp: "Formatted at 2026-06-21 06:57:17 on dist-test-slave-gmf2"
server_key: "63ca3ef2ac345b65778b863fb4ebda19"
server_key_iv: "156b0a85f4287080086ffe230077f2dd"
server_key_version: "encryptionkey@0"
I20260621 06:57:17.620398 10411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "5fed1a26e56c4c2ea5df22756070b42e"
format_stamp: "Formatted at 2026-06-21 06:57:17 on dist-test-slave-gmf2"
server_key: "63ca3ef2ac345b65778b863fb4ebda19"
server_key_iv: "156b0a85f4287080086ffe230077f2dd"
server_key_version: "encryptionkey@0"
I20260621 06:57:17.625561 10411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 06:57:17.628916 10426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:17.630321 10411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 06:57:17.630497 10411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "5fed1a26e56c4c2ea5df22756070b42e"
format_stamp: "Formatted at 2026-06-21 06:57:17 on dist-test-slave-gmf2"
server_key: "63ca3ef2ac345b65778b863fb4ebda19"
server_key_iv: "156b0a85f4287080086ffe230077f2dd"
server_key_version: "encryptionkey@0"
I20260621 06:57:17.630697 10411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:17.636045 10411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:17.639851 10411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:17.640081 10411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:17.658543 10411 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:35887
I20260621 06:57:17.658573 10477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:35887 every 8 connection(s)
I20260621 06:57:17.659927 10411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:17.661455 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 10411
I20260621 06:57:17.661633 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:17.662155 20963 external_mini_cluster.cc:1428] Setting key 49e014d8861e714f5da1ac159ec1f033
I20260621 06:57:17.664162 10478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:17.669839 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:17.663988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53023 (local address 127.20.120.254:35887)
0621 06:57:17.664775 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:17.664788 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:17.664827 (+ 39us) server_negotiation.cc:407] Connection header received
0621 06:57:17.666488 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:17.666509 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:17.666849 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:17.667181 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:17.668288 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:17.668294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:17.668306 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:17.668319 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:17.668392 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:17.668410 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:17.668571 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:17.668709 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 06:57:17.668909 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":358,"threads_started":1}
I20260621 06:57:17.673640 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:17.676460 10478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e: Bootstrap starting.
I20260621 06:57:17.679787 10478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:17.680751 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:17.678078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42969 (local address 127.20.120.254:35887)
0621 06:57:17.678300 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:17.678313 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:17.678796 (+ 483us) server_negotiation.cc:407] Connection header received
0621 06:57:17.678984 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:17.679002 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:17.679059 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:17.679152 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:17.680111 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:17.680118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:17.680122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:17.680143 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:17.680190 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:17.680205 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:17.680319 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:17.680415 (+ 96us) server_negotiation.cc:299] Negotiation successful
0621 06:57:17.680473 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 06:57:17.680963 10478 log.cc:824] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:17.683717 10478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e: No bootstrap required, opened a new log
I20260621 06:57:17.691007 10478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fed1a26e56c4c2ea5df22756070b42e" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 35887 } }
I20260621 06:57:17.691365 10478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:17.691468 10478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fed1a26e56c4c2ea5df22756070b42e, State: Initialized, Role: FOLLOWER
I20260621 06:57:17.692090 10478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fed1a26e56c4c2ea5df22756070b42e" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 35887 } }
I20260621 06:57:17.692345 10478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:17.692507 10478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:17.692687 10478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:17.694859 10478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fed1a26e56c4c2ea5df22756070b42e" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 35887 } }
I20260621 06:57:17.695322 10478 leader_election.cc:302] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fed1a26e56c4c2ea5df22756070b42e; no voters:
I20260621 06:57:17.695861 10478 leader_election.cc:288] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:17.695986 10488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:17.696686 10488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [term 1 LEADER]: Becoming Leader. State: Replica: 5fed1a26e56c4c2ea5df22756070b42e, State: Running, Role: LEADER
I20260621 06:57:17.697297 10488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fed1a26e56c4c2ea5df22756070b42e" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 35887 } }
I20260621 06:57:17.697844 10478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:17.700153 10490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fed1a26e56c4c2ea5df22756070b42e. Latest consensus state: current_term: 1 leader_uuid: "5fed1a26e56c4c2ea5df22756070b42e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fed1a26e56c4c2ea5df22756070b42e" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 35887 } } }
I20260621 06:57:17.700625 10490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:17.700585 10489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fed1a26e56c4c2ea5df22756070b42e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fed1a26e56c4c2ea5df22756070b42e" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 35887 } } }
I20260621 06:57:17.700923 10489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:17.702127 10496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:17.707322 10496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:17.715878 10496 catalog_manager.cc:1347] Generated new cluster ID: 90cc9526efe54d5cbaee5925f183aa28
I20260621 06:57:17.715989 10496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:17.728307 10496 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:17.728828 10496 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:17.744099 10496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fed1a26e56c4c2ea5df22756070b42e: Generated new TSK 0
I20260621 06:57:17.745034 10496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:17.760751 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10411
W20260621 06:57:17.761831 10473 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53023
W20260621 06:57:17.762539 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10411: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:57:17.762694 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 10411
2026-06-21T06:57:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1855 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:17 dist-test-slave-gmf2 krb5kdc[10509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:17 dist-test-slave-gmf2 krb5kdc[10509](info): set up 2 sockets
Jun 21 06:57:17 dist-test-slave-gmf2 krb5kdc[10509](info): commencing operation
krb5kdc: starting...
W20260621 06:57:18.882467 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:19 dist-test-slave-gmf2 krb5kdc[10509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:57:19.122327 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:40801
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:40801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:19.455276 10522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:19.455649 10522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:19.455832 10522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:19.463104 10522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:19.463272 10522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:19.463384 10522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:19.463505 10522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:19.463585 10522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:19.474258 10522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:40801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:40801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:19.475575 10522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:19.476926 10522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:19.488924 10527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:19.488863 10530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:19.488879 10528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:19.489959 10522 server_base.cc:1034] running on GCE node
I20260621 06:57:19.490693 10522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:19.491940 10522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:19.493149 10522 hybrid_clock.cc:648] HybridClock initialized: now 1782025039493105 us; error 45 us; skew 500 ppm
I20260621 06:57:19.493468 10522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:19 dist-test-slave-gmf2 krb5kdc[10509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025039, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:19.496480 10522 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:57:19.497830 10522 webserver.cc:460] Webserver started at http://127.20.120.254:37807/ using document root <none> and password file <none>
I20260621 06:57:19.498471 10522 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:19.498564 10522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:19.498791 10522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:19.501479 10522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "4add577892fc4cc7a6b190614b9ce704"
format_stamp: "Formatted at 2026-06-21 06:57:19 on dist-test-slave-gmf2"
server_key: "3255a6e1af57d42eb8fd3b429a6c747a"
server_key_iv: "f028771cc32078a15bbcb602c7e3e427"
server_key_version: "encryptionkey@0"
I20260621 06:57:19.502225 10522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "4add577892fc4cc7a6b190614b9ce704"
format_stamp: "Formatted at 2026-06-21 06:57:19 on dist-test-slave-gmf2"
server_key: "3255a6e1af57d42eb8fd3b429a6c747a"
server_key_iv: "f028771cc32078a15bbcb602c7e3e427"
server_key_version: "encryptionkey@0"
I20260621 06:57:19.506901 10522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 06:57:19.510027 10537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:19.511238 10522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 06:57:19.511404 10522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "4add577892fc4cc7a6b190614b9ce704"
format_stamp: "Formatted at 2026-06-21 06:57:19 on dist-test-slave-gmf2"
server_key: "3255a6e1af57d42eb8fd3b429a6c747a"
server_key_iv: "f028771cc32078a15bbcb602c7e3e427"
server_key_version: "encryptionkey@0"
I20260621 06:57:19.511580 10522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:19.534893 10522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:19.538940 10522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:19.539206 10522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:19.556852 10522 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:40801
I20260621 06:57:19.556913 10588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:40801 every 8 connection(s)
I20260621 06:57:19.558233 10522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:19.561723 10589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:19.564877 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 10522
I20260621 06:57:19.565065 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:19.565564 20963 external_mini_cluster.cc:1428] Setting key 187f8ccb857dfe0492d71168b0465e50
I20260621 06:57:19.575719 10589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704: Bootstrap starting.
I20260621 06:57:19.579257 10589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:19.580436 10589 log.cc:824] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:19.583425 10589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704: No bootstrap required, opened a new log
Jun 21 06:57:19 dist-test-slave-gmf2 krb5kdc[10509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:57:19.592872 10589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4add577892fc4cc7a6b190614b9ce704" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 40801 } }
I20260621 06:57:19.593277 10589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:19.593374 10589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4add577892fc4cc7a6b190614b9ce704, State: Initialized, Role: FOLLOWER
I20260621 06:57:19.594048 10589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4add577892fc4cc7a6b190614b9ce704" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 40801 } }
I20260621 06:57:19.594326 10589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:19.594466 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:19.567487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41983 (local address 127.20.120.254:40801)
0621 06:57:19.572924 (+ 5437us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:19.572943 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:19.573369 (+ 426us) server_negotiation.cc:407] Connection header received
0621 06:57:19.575781 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:19.575808 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:19.576282 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:19.576697 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:19.577958 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:19.579967 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:19.581515 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:19.582462 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:19.582662 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:19.586486 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:19.586496 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:19.586512 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:19.586562 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:19.590412 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:19.591006 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:19.591015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:19.591022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:19.591240 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:19.591567 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:19.591572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:19.591575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:19.592197 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:19.592463 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:19.592738 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 06:57:19.593147 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5180,"thread_start_us":4931,"threads_started":1}
I20260621 06:57:19.594537 10589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:19.594707 10589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:19.597478 10589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4add577892fc4cc7a6b190614b9ce704" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 40801 } }
I20260621 06:57:19.598003 10589 leader_election.cc:302] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4add577892fc4cc7a6b190614b9ce704; no voters:
I20260621 06:57:19.598197 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:19.598634 10589 leader_election.cc:288] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:19.598748 10594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:19.600431 10589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:19.607137 10594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [term 1 LEADER]: Becoming Leader. State: Replica: 4add577892fc4cc7a6b190614b9ce704, State: Running, Role: LEADER
I20260621 06:57:19.607900 10594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4add577892fc4cc7a6b190614b9ce704" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 40801 } }
I20260621 06:57:19.612269 10595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4add577892fc4cc7a6b190614b9ce704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4add577892fc4cc7a6b190614b9ce704" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 40801 } } }
I20260621 06:57:19.612633 10595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:19.613129 10596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4add577892fc4cc7a6b190614b9ce704. Latest consensus state: current_term: 1 leader_uuid: "4add577892fc4cc7a6b190614b9ce704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4add577892fc4cc7a6b190614b9ce704" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 40801 } } }
I20260621 06:57:19.613385 10596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:19.620201 10613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:19.625165 10613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:19.628412 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:19.609808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38745 (local address 127.20.120.254:40801)
0621 06:57:19.610063 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:19.610071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:19.610550 (+ 479us) server_negotiation.cc:407] Connection header received
0621 06:57:19.610776 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:19.610784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:19.610882 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:19.611007 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:19.613095 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:19.614299 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:19.617042 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:19.617936 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:19.618050 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:19.621564 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:19.621569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:19.621572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:19.621598 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:19.625412 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:19.626207 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:19.626212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:19.626216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:19.626349 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:19.626782 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:19.626787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:19.626791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:19.627213 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:19.627462 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:19.627810 (+ 348us) server_negotiation.cc:299] Negotiation successful
0621 06:57:19.628023 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260621 06:57:19.633785 10613 catalog_manager.cc:1347] Generated new cluster ID: e5049a91c3454422b44752eb0d294ff0
I20260621 06:57:19.633888 10613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:19.653759 10613 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:19.655210 10613 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:19.669232 10613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4add577892fc4cc7a6b190614b9ce704: Generated new TSK 0
I20260621 06:57:19.670063 10613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:19.681241 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10522
I20260621 06:57:19.682636 10584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:57:19.773813 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 10522
2026-06-21T06:57:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:19 dist-test-slave-gmf2 krb5kdc[10622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:19 dist-test-slave-gmf2 krb5kdc[10622](info): set up 2 sockets
Jun 21 06:57:19 dist-test-slave-gmf2 krb5kdc[10622](info): commencing operation
krb5kdc: starting...
W20260621 06:57:20.866381 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:21 dist-test-slave-gmf2 krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:57:21.099076 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:43899
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:35073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:43899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:21.417035 10635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:21.417273 10635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:21.417357 10635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:21.424105 10635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:21.424203 10635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:21.424247 10635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:21.424305 10635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:21.424347 10635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:21.433377 10635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:35073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:43899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:43899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:21.434569 10635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:21.436434 10635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:21.448803 10641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:21.448849 10643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:21.448870 10640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:21.449698 10635 server_base.cc:1034] running on GCE node
I20260621 06:57:21.450380 10635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:21.451669 10635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:21.452986 10635 hybrid_clock.cc:648] HybridClock initialized: now 1782025041452926 us; error 43 us; skew 500 ppm
I20260621 06:57:21.453274 10635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:21 dist-test-slave-gmf2 krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025041, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:21.456146 10635 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:57:21.457435 10635 webserver.cc:460] Webserver started at http://127.20.120.254:37741/ using document root <none> and password file <none>
I20260621 06:57:21.458076 10635 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:21.458209 10635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:21.458475 10635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:21.461239 10635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "e782fb47c3434fbe85815279b5b18964"
format_stamp: "Formatted at 2026-06-21 06:57:21 on dist-test-slave-gmf2"
server_key: "958c9c522fd6764fd112849bdd817487"
server_key_iv: "a1f21068c3b3483c3f025ea577c24ed9"
server_key_version: "encryptionkey@0"
I20260621 06:57:21.462028 10635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "e782fb47c3434fbe85815279b5b18964"
format_stamp: "Formatted at 2026-06-21 06:57:21 on dist-test-slave-gmf2"
server_key: "958c9c522fd6764fd112849bdd817487"
server_key_iv: "a1f21068c3b3483c3f025ea577c24ed9"
server_key_version: "encryptionkey@0"
I20260621 06:57:21.466866 10635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 06:57:21.470029 10650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:21.471185 10635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 06:57:21.471350 10635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "e782fb47c3434fbe85815279b5b18964"
format_stamp: "Formatted at 2026-06-21 06:57:21 on dist-test-slave-gmf2"
server_key: "958c9c522fd6764fd112849bdd817487"
server_key_iv: "a1f21068c3b3483c3f025ea577c24ed9"
server_key_version: "encryptionkey@0"
I20260621 06:57:21.471527 10635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:21.498769 10635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:21.502501 10635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:21.502776 10635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:21.520721 10635 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:43899
I20260621 06:57:21.520741 10701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:43899 every 8 connection(s)
I20260621 06:57:21.522054 10635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:21.525431 10702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:21.532287 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 10635
I20260621 06:57:21.532466 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:21.532992 20963 external_mini_cluster.cc:1428] Setting key bfa6b67805fc5c65fb38aeb1f7ab5ead
I20260621 06:57:21.535863 10702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964: Bootstrap starting.
I20260621 06:57:21.538895 10702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:21.539954 10702 log.cc:824] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:21.542760 10702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964: No bootstrap required, opened a new log
I20260621 06:57:21.545234 10705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:21.534649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42461 (local address 127.20.120.254:43899)
0621 06:57:21.535399 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:21.535409 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:21.535446 (+ 37us) server_negotiation.cc:407] Connection header received
0621 06:57:21.537080 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:21.537102 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:21.537467 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:21.537745 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:21.538312 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:21.539557 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:21.541197 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:21.541802 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:21.542003 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:21.543121 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:21.543129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:21.543139 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:21.543617 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:21.543685 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:21.543707 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:21.543792 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:21.544045 (+ 253us) server_negotiation.cc:299] Negotiation successful
0621 06:57:21.544315 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":352,"threads_started":1}
I20260621 06:57:21.548846 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:21.551928 10702 raft_consensus.cc:348] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e782fb47c3434fbe85815279b5b18964" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 43899 } }
I20260621 06:57:21.552408 10702 raft_consensus.cc:374] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:21.552523 10702 raft_consensus.cc:729] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e782fb47c3434fbe85815279b5b18964, State: Initialized, Role: FOLLOWER
I20260621 06:57:21.553179 10702 consensus_queue.cc:260] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e782fb47c3434fbe85815279b5b18964" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 43899 } }
I20260621 06:57:21.553465 10702 raft_consensus.cc:388] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:21.553623 10702 raft_consensus.cc:482] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:21.553810 10702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:21.556737 10702 raft_consensus.cc:504] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e782fb47c3434fbe85815279b5b18964" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 43899 } }
I20260621 06:57:21.557291 10702 leader_election.cc:302] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e782fb47c3434fbe85815279b5b18964; no voters:
I20260621 06:57:21.557895 10702 leader_election.cc:288] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:21.558044 10712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:21.559286 10712 raft_consensus.cc:686] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [term 1 LEADER]: Becoming Leader. State: Replica: e782fb47c3434fbe85815279b5b18964, State: Running, Role: LEADER
I20260621 06:57:21.559927 10712 consensus_queue.cc:237] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e782fb47c3434fbe85815279b5b18964" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 43899 } }
I20260621 06:57:21.560236 10702 sys_catalog.cc:564] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:21.561316 10705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:21.553881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36627 (local address 127.20.120.254:43899)
0621 06:57:21.554072 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:21.554080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:21.554607 (+ 527us) server_negotiation.cc:407] Connection header received
0621 06:57:21.554781 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:21.554785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:21.554844 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:21.554919 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:21.555461 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:21.556120 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:21.558129 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:21.558962 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:21.559115 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:21.560506 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:21.560512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:21.560516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:21.560542 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:21.560585 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:21.560604 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:21.560719 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:21.560891 (+ 172us) server_negotiation.cc:299] Negotiation successful
0621 06:57:21.560974 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 06:57:21.562924 10713 sys_catalog.cc:455] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e782fb47c3434fbe85815279b5b18964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e782fb47c3434fbe85815279b5b18964" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 43899 } } }
I20260621 06:57:21.563278 10713 sys_catalog.cc:458] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:21.566167 10714 sys_catalog.cc:455] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e782fb47c3434fbe85815279b5b18964. Latest consensus state: current_term: 1 leader_uuid: "e782fb47c3434fbe85815279b5b18964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e782fb47c3434fbe85815279b5b18964" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 43899 } } }
I20260621 06:57:21.566599 10714 sys_catalog.cc:458] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:21.567404 10717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:21.573413 10717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:21.581627 10717 catalog_manager.cc:1347] Generated new cluster ID: 1b29fb2558204ed2a627c189811298ff
I20260621 06:57:21.581732 10717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:21.601315 10717 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:21.602963 10717 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:21.615464 10717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e782fb47c3434fbe85815279b5b18964: Generated new TSK 0
I20260621 06:57:21.616189 10717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:21.622592 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10635
W20260621 06:57:21.623531 10697 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42461
W20260621 06:57:21.624166 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10635: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:57:21.624315 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 10635
2026-06-21T06:57:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1849 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:21 dist-test-slave-gmf2 krb5kdc[10733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:21 dist-test-slave-gmf2 krb5kdc[10733](info): set up 2 sockets
Jun 21 06:57:21 dist-test-slave-gmf2 krb5kdc[10733](info): commencing operation
krb5kdc: starting...
W20260621 06:57:22.710409 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:22 dist-test-slave-gmf2 krb5kdc[10733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:57:22.940321 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:40011
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:40011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:23.257843 10746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:23.258206 10746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:23.258363 10746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:23.266043 10746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:23.266134 10746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:23.266176 10746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:23.266228 10746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:23.266260 10746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:23.275804 10746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:40011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:40011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:23.277071 10746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:23.278362 10746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:23.289824 10751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:23.290198 10754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:23.290460 10752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:23.290963 10746 server_base.cc:1034] running on GCE node
I20260621 06:57:23.291738 10746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:23.293076 10746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:23.294322 10746 hybrid_clock.cc:648] HybridClock initialized: now 1782025043294261 us; error 40 us; skew 500 ppm
I20260621 06:57:23.294705 10746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:23 dist-test-slave-gmf2 krb5kdc[10733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025043, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:23.300969 10746 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:57:23.302510 10746 webserver.cc:460] Webserver started at http://127.20.120.254:45463/ using document root <none> and password file <none>
I20260621 06:57:23.303192 10746 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:23.303335 10746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:23.303632 10746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:23.306615 10746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "554e8b32ae794423bf6ac454f49d2ca4"
format_stamp: "Formatted at 2026-06-21 06:57:23 on dist-test-slave-gmf2"
server_key: "3cd01a5c562f0bb44b8105506c8a381f"
server_key_iv: "0b2ebcb0d38415f9a8cfd0863ec6df34"
server_key_version: "encryptionkey@0"
I20260621 06:57:23.307365 10746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "554e8b32ae794423bf6ac454f49d2ca4"
format_stamp: "Formatted at 2026-06-21 06:57:23 on dist-test-slave-gmf2"
server_key: "3cd01a5c562f0bb44b8105506c8a381f"
server_key_iv: "0b2ebcb0d38415f9a8cfd0863ec6df34"
server_key_version: "encryptionkey@0"
I20260621 06:57:23.312201 10746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 06:57:23.315378 10761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:23.316572 10746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 06:57:23.316747 10746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "554e8b32ae794423bf6ac454f49d2ca4"
format_stamp: "Formatted at 2026-06-21 06:57:23 on dist-test-slave-gmf2"
server_key: "3cd01a5c562f0bb44b8105506c8a381f"
server_key_iv: "0b2ebcb0d38415f9a8cfd0863ec6df34"
server_key_version: "encryptionkey@0"
I20260621 06:57:23.316933 10746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:23.322513 10746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:23.326437 10746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:23.326725 10746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:23.345496 10746 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:40011
I20260621 06:57:23.345532 10812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:40011 every 8 connection(s)
I20260621 06:57:23.346884 10746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:23.349882 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 10746
I20260621 06:57:23.350099 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:23.350486 10813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:23.350562 20963 external_mini_cluster.cc:1428] Setting key 16fa30767c05219e61ab2f7a46a01235
I20260621 06:57:23.358909 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:23.352245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33965 (local address 127.20.120.254:40011)
0621 06:57:23.353088 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:23.353102 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:23.353154 (+ 52us) server_negotiation.cc:407] Connection header received
0621 06:57:23.355241 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:23.355268 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:23.355684 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:23.356101 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:23.357063 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:23.357076 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:23.357092 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:23.357114 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:23.357182 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:23.357211 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:23.357419 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:23.357549 (+ 130us) server_negotiation.cc:299] Negotiation successful
0621 06:57:23.357740 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":366,"threads_started":1}
I20260621 06:57:23.362305 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:23.364471 10813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4: Bootstrap starting.
I20260621 06:57:23.367955 10813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:23.368908 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:23.366555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34109 (local address 127.20.120.254:40011)
0621 06:57:23.366750 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:23.366756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:23.367095 (+ 339us) server_negotiation.cc:407] Connection header received
0621 06:57:23.367207 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:23.367212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:23.367272 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:23.367374 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:23.368288 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:23.368294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:23.368300 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 06:57:23.368326 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:23.368377 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 06:57:23.368400 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:23.368526 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:23.368594 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 06:57:23.368653 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 06:57:23.369288 10813 log.cc:824] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:23.371783 10813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4: No bootstrap required, opened a new log
I20260621 06:57:23.378851 10813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8b32ae794423bf6ac454f49d2ca4" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 40011 } }
I20260621 06:57:23.379153 10813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 06:57:23.379258 10813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554e8b32ae794423bf6ac454f49d2ca4, State: Initialized, Role: FOLLOWER
I20260621 06:57:23.379732 10813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8b32ae794423bf6ac454f49d2ca4" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 40011 } }
I20260621 06:57:23.379941 10813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 06:57:23.380087 10813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 06:57:23.380239 10813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:23.382335 10813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8b32ae794423bf6ac454f49d2ca4" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 40011 } }
I20260621 06:57:23.382786 10813 leader_election.cc:302] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 554e8b32ae794423bf6ac454f49d2ca4; no voters:
I20260621 06:57:23.383289 10813 leader_election.cc:288] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 06:57:23.383451 10823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:23.384646 10823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [term 1 LEADER]: Becoming Leader. State: Replica: 554e8b32ae794423bf6ac454f49d2ca4, State: Running, Role: LEADER
I20260621 06:57:23.385124 10823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8b32ae794423bf6ac454f49d2ca4" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 40011 } }
I20260621 06:57:23.385275 10813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:23.387890 10825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 554e8b32ae794423bf6ac454f49d2ca4. Latest consensus state: current_term: 1 leader_uuid: "554e8b32ae794423bf6ac454f49d2ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8b32ae794423bf6ac454f49d2ca4" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 40011 } } }
I20260621 06:57:23.387852 10824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "554e8b32ae794423bf6ac454f49d2ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554e8b32ae794423bf6ac454f49d2ca4" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 40011 } } }
I20260621 06:57:23.388269 10825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:23.388317 10824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:23.390116 10831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:23.395162 10831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:23.403403 10831 catalog_manager.cc:1347] Generated new cluster ID: 8ef2bd460de74c579f8597ba1ec7f8eb
I20260621 06:57:23.403532 10831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:23.417394 10831 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:23.417771 10831 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:23.427536 10831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 554e8b32ae794423bf6ac454f49d2ca4: Generated new TSK 0
I20260621 06:57:23.428253 10831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 06:57:23.436661 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10746
W20260621 06:57:23.438000 10808 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33965
W20260621 06:57:23.438830 20963 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10746: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 06:57:23.438982 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 10746
2026-06-21T06:57:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1820 ms)
[----------] 18 tests from AuthTokenIssuingTest (34211 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:23 dist-test-slave-gmf2 krb5kdc[10844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:23 dist-test-slave-gmf2 krb5kdc[10844](info): set up 2 sockets
Jun 21 06:57:23 dist-test-slave-gmf2 krb5kdc[10844](info): commencing operation
krb5kdc: starting...
W20260621 06:57:24.530488 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:24 dist-test-slave-gmf2 krb5kdc[10844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:57:24.782123 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:45907
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:45907,127.20.120.253:45387,127.20.120.252:36887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:25.102847 10857 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 06:57:25.103202 10857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:25.103298 10857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:25.103428 10857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:25.110517 10857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:25.110600 10857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:25.110700 10857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:25.110766 10857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:25.110873 10857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:25.120097 10857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:45907,127.20.120.253:45387,127.20.120.252:36887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:45907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:25.121381 10857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:25.123026 10857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:25.135093 10862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:25.135188 10865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:25.135355 10863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:25.135952 10857 server_base.cc:1034] running on GCE node
I20260621 06:57:25.136693 10857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:25.138072 10857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:25.139312 10857 hybrid_clock.cc:648] HybridClock initialized: now 1782025045139261 us; error 50 us; skew 500 ppm
I20260621 06:57:25.139637 10857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:25 dist-test-slave-gmf2 krb5kdc[10844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025045, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:25.142714 10857 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:57:25.144071 10857 webserver.cc:460] Webserver started at http://127.20.120.254:44733/ using document root <none> and password file <none>
I20260621 06:57:25.144728 10857 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:25.144811 10857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:25.145148 10857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:25.148077 10857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "9caed4c752b5471ea799665f2513c75c"
format_stamp: "Formatted at 2026-06-21 06:57:25 on dist-test-slave-gmf2"
server_key: "412b8bb1643574b6aa0c25b07036f59d"
server_key_iv: "1cd260a70f1337e3d7f858c7ffb4c69d"
server_key_version: "encryptionkey@0"
I20260621 06:57:25.148799 10857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "9caed4c752b5471ea799665f2513c75c"
format_stamp: "Formatted at 2026-06-21 06:57:25 on dist-test-slave-gmf2"
server_key: "412b8bb1643574b6aa0c25b07036f59d"
server_key_iv: "1cd260a70f1337e3d7f858c7ffb4c69d"
server_key_version: "encryptionkey@0"
I20260621 06:57:25.153617 10857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 06:57:25.157135 10872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:25.158540 10857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 06:57:25.158715 10857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "9caed4c752b5471ea799665f2513c75c"
format_stamp: "Formatted at 2026-06-21 06:57:25 on dist-test-slave-gmf2"
server_key: "412b8bb1643574b6aa0c25b07036f59d"
server_key_iv: "1cd260a70f1337e3d7f858c7ffb4c69d"
server_key_version: "encryptionkey@0"
I20260621 06:57:25.158895 10857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:25.175772 10857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:25.179693 10857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:25.179980 10857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:25.198580 10857 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:45907
I20260621 06:57:25.198621 10923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:45907 every 8 connection(s)
I20260621 06:57:25.199934 10857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:25.203871 10924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:25.206372 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 10857
I20260621 06:57:25.206596 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:25.207458 20963 external_mini_cluster.cc:1428] Setting key 6b01a19b4e1f5e9c80260f9a5a1cdfb7
I20260621 06:57:25.213681 10924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:57:25 dist-test-slave-gmf2 krb5kdc[10844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025045, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.120.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.253@KRBTEST.COM" created.
I20260621 06:57:25.239841 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.217967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.20.120.254:45907)
0621 06:57:25.219949 (+ 1982us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:25.219959 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:25.220012 (+ 53us) server_negotiation.cc:407] Connection header received
0621 06:57:25.221566 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:25.221577 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:25.221902 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:25.222125 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:25.223420 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.225343 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.226845 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.227784 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.227985 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:25.231557 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:25.231569 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:25.231594 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:25.231638 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:25.235559 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:25.236386 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:25.236392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:25.236394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:25.236534 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:25.236904 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:25.236907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:25.236909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:25.237425 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:25.237609 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:25.238351 (+ 742us) server_negotiation.cc:299] Negotiation successful
0621 06:57:25.238709 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":1752,"spinlock_wait_cycles":45440,"thread_start_us":1573,"threads_started":1}
I20260621 06:57:25.239893 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.217967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45907 (local address 127.0.0.1:42508)
0621 06:57:25.218959 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.219012 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:25.221176 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:25.222176 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:25.222181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:25.222832 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:25.223202 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:25.223220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.225511 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.225515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:25.226701 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:25.226715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.228016 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.228022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:25.228149 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:25.228854 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:25.228901 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:25.231322 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:25.235711 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:25.235721 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:25.235748 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:25.236251 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:25.236641 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:25.236645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:25.236647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:25.236800 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:25.237579 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:25.237585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:25.237817 (+ 232us) client_negotiation.cc:769] Sending connection context
0621 06:57:25.238229 (+ 412us) client_negotiation.cc:240] Negotiation successful
0621 06:57:25.238880 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":100,"thread_start_us":423,"threads_started":1}
I20260621 06:57:25.242767 10924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } has no permanent_uuid. Determining permanent_uuid...
I20260621 06:57:25.243753 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.243398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46808)
0621 06:57:25.243557 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.243654 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260621 06:57:25.243999 10876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
I20260621 06:57:25.245529 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.245195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46824)
0621 06:57:25.245372 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.245430 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 06:57:25.245728 10924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:45387: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.253.keytab.
Entry for principal kudu/127.20.120.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.253.keytab.
I20260621 06:57:25.267452 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.253:45387
--webserver_interface=127.20.120.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.253.keytab
--principal=kudu/127.20.120.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:45907,127.20.120.253:45387,127.20.120.252:36887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:57:25.295020 10924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } attempt: 1
I20260621 06:57:25.296202 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.295755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46828)
0621 06:57:25.295948 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.296054 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260621 06:57:25.297127 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.296813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46840)
0621 06:57:25.296936 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.297002 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 06:57:25.297350 10924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:45387: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
I20260621 06:57:25.365579 10924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } attempt: 2
I20260621 06:57:25.366786 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.366281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46842)
0621 06:57:25.366529 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.366621 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260621 06:57:25.367789 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.367455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46844)
0621 06:57:25.367590 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.367667 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 06:57:25.367981 10924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:45387: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
I20260621 06:57:25.459214 10924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } attempt: 3
I20260621 06:57:25.460404 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.459999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46856)
0621 06:57:25.460196 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.460280 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 06:57:25.461529 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.461221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46858)
0621 06:57:25.461346 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.461408 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 06:57:25.461767 10924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:45387: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
W20260621 06:57:25.603569 10931 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 06:57:25.603853 10931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:25.603960 10931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:25.604096 10931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260621 06:57:25.605003 10924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } attempt: 4
I20260621 06:57:25.606271 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.605685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46408)
0621 06:57:25.606031 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.606122 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260621 06:57:25.607218 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.606885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46412)
0621 06:57:25.607029 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.607107 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 06:57:25.607417 10924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:45387: Network error: Client connection negotiation failed: client connection to 127.20.120.253:45387: connect: Connection refused (error 111)
W20260621 06:57:25.611797 10931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:25.611894 10931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:25.611969 10931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:25.612042 10931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:25.612152 10931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:25.621368 10931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:45907,127.20.120.253:45387,127.20.120.252:36887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.253.keytab
--principal=kudu/127.20.120.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.253:45387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.120.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:25.622610 10931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:25.623876 10931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:25.635438 10937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:25.635389 10936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:25.635471 10939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:25.636123 10931 server_base.cc:1034] running on GCE node
I20260621 06:57:25.636875 10931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:25.638233 10931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:25.639472 10931 hybrid_clock.cc:648] HybridClock initialized: now 1782025045639413 us; error 36 us; skew 500 ppm
I20260621 06:57:25.639779 10931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:25 dist-test-slave-gmf2 krb5kdc[10844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025045, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:25.642807 10931 init.cc:377] Logged in from keytab as kudu/127.20.120.253@KRBTEST.COM (short username kudu)
I20260621 06:57:25.644172 10931 webserver.cc:460] Webserver started at http://127.20.120.253:45603/ using document root <none> and password file <none>
I20260621 06:57:25.644821 10931 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:25.644953 10931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:25.645253 10931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:25.648087 10931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/data/instance:
uuid: "8cf5361404f94642a6d48eac5dcd2b71"
format_stamp: "Formatted at 2026-06-21 06:57:25 on dist-test-slave-gmf2"
server_key: "ca47f782243ffd2f1a82eea350d91112"
server_key_iv: "43f382f187a9317ca2c71b2f8e36459a"
server_key_version: "encryptionkey@0"
I20260621 06:57:25.648862 10931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/wal/instance:
uuid: "8cf5361404f94642a6d48eac5dcd2b71"
format_stamp: "Formatted at 2026-06-21 06:57:25 on dist-test-slave-gmf2"
server_key: "ca47f782243ffd2f1a82eea350d91112"
server_key_iv: "43f382f187a9317ca2c71b2f8e36459a"
server_key_version: "encryptionkey@0"
I20260621 06:57:25.653501 10931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260621 06:57:25.656731 10946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:25.658004 10931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 06:57:25.658172 10931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/wal
uuid: "8cf5361404f94642a6d48eac5dcd2b71"
format_stamp: "Formatted at 2026-06-21 06:57:25 on dist-test-slave-gmf2"
server_key: "ca47f782243ffd2f1a82eea350d91112"
server_key_iv: "43f382f187a9317ca2c71b2f8e36459a"
server_key_version: "encryptionkey@0"
I20260621 06:57:25.658376 10931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:25.676426 10931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:25.680599 10931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:25.680894 10931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:25.699035 10931 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.253:45387
I20260621 06:57:25.699054 10997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.253:45387 every 8 connection(s)
I20260621 06:57:25.700259 10931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
I20260621 06:57:25.701174 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 10931
I20260621 06:57:25.701356 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-1/wal/instance
I20260621 06:57:25.701846 20963 external_mini_cluster.cc:1428] Setting key e06ddda80e15d70530a8c4897af33b38
I20260621 06:57:25.704754 10998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:25.713546 10998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:57:25 dist-test-slave-gmf2 krb5kdc[10844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025045, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.253@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.120.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.252@KRBTEST.COM" created.
I20260621 06:57:25.736706 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.716081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.20.120.254:45907)
0621 06:57:25.716313 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:25.716321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:25.717349 (+ 1028us) server_negotiation.cc:407] Connection header received
0621 06:57:25.720159 (+ 2810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:25.720162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:25.720230 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:25.720313 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:25.722406 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.723118 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.725537 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.725996 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.726090 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:25.730257 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:25.730267 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:25.730274 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:25.730320 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:25.733565 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:25.734345 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:25.734354 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:25.734360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:25.734519 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:25.734897 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:25.734905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:25.734910 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:25.735280 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:25.735451 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:25.736039 (+ 588us) server_negotiation.cc:299] Negotiation successful
0621 06:57:25.736273 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 06:57:25.737251 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.716434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45907 (local address 127.0.0.1:36620)
0621 06:57:25.717169 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.717220 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:25.719868 (+ 2648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:25.720584 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:25.720597 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:25.721399 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:25.722240 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:25.722266 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.723288 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.723296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:25.725396 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:25.725414 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.726223 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.726233 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:25.726593 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:25.727337 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:25.727366 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:25.730005 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:25.733705 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:25.733711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:25.733721 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:25.734203 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:25.734625 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:25.734628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:25.734630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:25.734779 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:25.735394 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:25.735399 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:25.735691 (+ 292us) client_negotiation.cc:769] Sending connection context
0621 06:57:25.735884 (+ 193us) client_negotiation.cc:240] Negotiation successful
0621 06:57:25.736200 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":276,"threads_started":1}
I20260621 06:57:25.738873 10998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:57:25 dist-test-slave-gmf2 krb5kdc[10844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025045, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.253@KRBTEST.COM for kudu/127.20.120.253@KRBTEST.COM
I20260621 06:57:25.754912 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.739477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46418)
0621 06:57:25.739640 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.739661 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:25.739768 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:25.740799 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:25.740807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:25.741282 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:25.741487 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:25.741500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.742838 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.742846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:25.744359 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:25.744374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.745218 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.745227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:25.745462 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:25.746134 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:25.746158 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:25.748425 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:25.752578 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:25.752581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:25.752584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:25.752927 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:25.753376 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:25.753379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:25.753381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:25.753526 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:25.754278 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:25.754281 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:25.754386 (+ 105us) client_negotiation.cc:769] Sending connection context
0621 06:57:25.754495 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 06:57:25.754667 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 06:57:25.755354 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.739751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.20.120.253:45387)
0621 06:57:25.740389 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:25.740395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:25.740419 (+ 24us) server_negotiation.cc:407] Connection header received
0621 06:57:25.740460 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:25.740465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:25.740638 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:25.740750 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:25.741655 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.742709 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.744499 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.745071 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.745168 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:25.748653 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:25.748667 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:25.748684 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:25.748727 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:25.752449 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:25.753074 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:25.753085 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:25.753090 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:25.753243 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:25.753649 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:25.753654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:25.753657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:25.754158 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:25.754377 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:25.754687 (+ 310us) server_negotiation.cc:299] Negotiation successful
0621 06:57:25.754941 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":364,"threads_started":1}
I20260621 06:57:25.757477 10998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } has no permanent_uuid. Determining permanent_uuid...
I20260621 06:57:25.758720 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.758308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58078)
0621 06:57:25.758478 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.758589 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260621 06:57:25.758988 10948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
I20260621 06:57:25.760464 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.760176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58084)
0621 06:57:25.760312 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.760378 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 06:57:25.760727 10998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36887: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.252.keytab.
Entry for principal kudu/127.20.120.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.252.keytab.
I20260621 06:57:25.764721 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.252:36887
--webserver_interface=127.20.120.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.252.keytab
--principal=kudu/127.20.120.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:45907,127.20.120.253:45387,127.20.120.252:36887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:57:25.810066 10998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } attempt: 1
I20260621 06:57:25.811338 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.810859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58094)
0621 06:57:25.811082 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.811195 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260621 06:57:25.812259 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.811939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58098)
0621 06:57:25.812101 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.812166 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 06:57:25.812448 10998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36887: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
I20260621 06:57:25.880651 10998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } attempt: 2
I20260621 06:57:25.881732 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.881293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58100)
0621 06:57:25.881481 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.881576 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260621 06:57:25.882711 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.882402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58112)
0621 06:57:25.882536 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.882613 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 06:57:25.882921 10998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36887: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
I20260621 06:57:25.906659 10924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } attempt: 5
Jun 21 06:57:25 dist-test-slave-gmf2 krb5kdc[10844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025045, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for kudu/127.20.120.253@KRBTEST.COM
I20260621 06:57:25.919615 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.907492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.20.120.253:45387)
0621 06:57:25.907693 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:25.907700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:25.907808 (+ 108us) server_negotiation.cc:407] Connection header received
0621 06:57:25.908023 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:25.908027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:25.908114 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:25.908221 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:25.909005 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.909696 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.910928 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.911366 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.911501 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:25.914582 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:25.914587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:25.914590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:25.914620 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:25.917098 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:25.917704 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:25.917711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:25.917714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:25.917836 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:25.918217 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:25.918223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:25.918227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:25.918491 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:25.918673 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:25.919052 (+ 379us) server_negotiation.cc:299] Negotiation successful
0621 06:57:25.919288 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 06:57:25.919617 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.907467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46430)
0621 06:57:25.907692 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.907712 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:25.907879 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:25.908257 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:25.908262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:25.908668 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:25.908846 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:25.908855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.909833 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.909840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:25.910778 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:25.910790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:25.911510 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:25.911515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:25.911740 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:25.912351 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:25.912369 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:25.914394 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:25.917233 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:25.917238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:25.917242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:25.917587 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:25.917965 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:25.917970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:25.917973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:25.918113 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:25.918623 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:25.918627 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:25.918798 (+ 171us) client_negotiation.cc:769] Sending connection context
0621 06:57:25.918942 (+ 144us) client_negotiation.cc:240] Negotiation successful
0621 06:57:25.919248 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 06:57:25.920722 10924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } has no permanent_uuid. Determining permanent_uuid...
I20260621 06:57:25.921772 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.921368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58118)
0621 06:57:25.921534 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.921631 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260621 06:57:25.922760 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.922445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58126)
0621 06:57:25.922589 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.922667 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 06:57:25.923009 10924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36887: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
I20260621 06:57:25.974123 10998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } attempt: 3
I20260621 06:57:25.974223 10924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } attempt: 1
I20260621 06:57:25.975325 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.974783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58130)
0621 06:57:25.975016 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.975134 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260621 06:57:25.975365 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.974835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58136)
0621 06:57:25.975105 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.975192 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260621 06:57:25.976402 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.975933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58150)
0621 06:57:25.976165 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.976256 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260621 06:57:25.976501 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:25.976129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58154)
0621 06:57:25.976299 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:25.976374 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260621 06:57:25.976647 10998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36887: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
W20260621 06:57:25.976768 10924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36887: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
I20260621 06:57:26.044965 10924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } attempt: 2
I20260621 06:57:26.046173 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.045664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58170)
0621 06:57:26.045868 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:26.046011 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260621 06:57:26.047120 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.046766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58174)
0621 06:57:26.046929 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:26.046999 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 06:57:26.047317 10924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36887: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
W20260621 06:57:26.090678 11005 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 06:57:26.091039 11005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:26.091171 11005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:26.091307 11005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:26.098906 11005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:26.099076 11005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:26.099222 11005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:26.099313 11005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:26.099426 11005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:26.109549 11005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:45907,127.20.120.253:45387,127.20.120.252:36887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.252.keytab
--principal=kudu/127.20.120.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.252:36887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.120.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:26.110890 11005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:26.112267 11005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 06:57:26.119853 10998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } attempt: 4
I20260621 06:57:26.120975 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.120522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58182)
0621 06:57:26.120741 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:26.120823 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260621 06:57:26.122012 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.121631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58184)
0621 06:57:26.121771 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:26.121843 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 06:57:26.122228 10998 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36887: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
W20260621 06:57:26.124377 11010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:26.124327 11013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:26.124337 11011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:26.125226 11005 server_base.cc:1034] running on GCE node
I20260621 06:57:26.126210 11005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:26.128075 11005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:26.129309 11005 hybrid_clock.cc:648] HybridClock initialized: now 1782025046129235 us; error 49 us; skew 500 ppm
I20260621 06:57:26.129676 11005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:26 dist-test-slave-gmf2 krb5kdc[10844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025046, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:26.132858 11005 init.cc:377] Logged in from keytab as kudu/127.20.120.252@KRBTEST.COM (short username kudu)
I20260621 06:57:26.134352 11005 webserver.cc:460] Webserver started at http://127.20.120.252:45263/ using document root <none> and password file <none>
I20260621 06:57:26.135032 11005 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:26.135164 11005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:26.135493 11005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:26.138422 11005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/data/instance:
uuid: "5ae13c8699b94a748e07d54ab3a978aa"
format_stamp: "Formatted at 2026-06-21 06:57:26 on dist-test-slave-gmf2"
server_key: "6a89364cb196f1d24957da98238f8368"
server_key_iv: "21634bf8b3a779b2f3e4b5408a0753b0"
server_key_version: "encryptionkey@0"
I20260621 06:57:26.139257 11005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/wal/instance:
uuid: "5ae13c8699b94a748e07d54ab3a978aa"
format_stamp: "Formatted at 2026-06-21 06:57:26 on dist-test-slave-gmf2"
server_key: "6a89364cb196f1d24957da98238f8368"
server_key_iv: "21634bf8b3a779b2f3e4b5408a0753b0"
server_key_version: "encryptionkey@0"
I20260621 06:57:26.144016 11005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 06:57:26.147223 11020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:26.148317 11005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 06:57:26.148483 11005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/wal
uuid: "5ae13c8699b94a748e07d54ab3a978aa"
format_stamp: "Formatted at 2026-06-21 06:57:26 on dist-test-slave-gmf2"
server_key: "6a89364cb196f1d24957da98238f8368"
server_key_iv: "21634bf8b3a779b2f3e4b5408a0753b0"
server_key_version: "encryptionkey@0"
I20260621 06:57:26.148669 11005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:26.153506 10924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } attempt: 3
I20260621 06:57:26.154574 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.154123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58186)
0621 06:57:26.154347 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:26.154422 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 06:57:26.155540 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.155173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58194)
0621 06:57:26.155375 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:26.155446 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260621 06:57:26.155762 10924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:36887: Network error: Client connection negotiation failed: client connection to 127.20.120.252:36887: connect: Connection refused (error 111)
I20260621 06:57:26.162657 11005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:26.166255 11005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:26.166492 11005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:26.184067 11005 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.252:36887
I20260621 06:57:26.184094 11071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.252:36887 every 8 connection(s)
I20260621 06:57:26.185336 11005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
I20260621 06:57:26.187615 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 11005
I20260621 06:57:26.187798 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782024700637160-20963-0/minicluster-data/master-2/wal/instance
I20260621 06:57:26.188330 20963 external_mini_cluster.cc:1428] Setting key 40a31c669bbcdbf8637df0b209a5a942
I20260621 06:57:26.189436 11072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:26.198619 11072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:57:26 dist-test-slave-gmf2 krb5kdc[10844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.252@KRBTEST.COM
Jun 21 06:57:26 dist-test-slave-gmf2 krb5kdc[10844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025046, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.252@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:57:26.212106 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.190168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.20.120.252:36887)
0621 06:57:26.190925 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.190936 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.190981 (+ 45us) server_negotiation.cc:407] Connection header received
0621 06:57:26.192425 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.192443 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.192840 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.193152 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.194938 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.196229 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.198075 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.199038 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.199228 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.204130 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:26.204145 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:26.204155 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:26.204192 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:26.207716 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.208405 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.208414 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.208421 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.208603 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.209104 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.209108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.209110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.209659 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:26.209980 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:26.210472 (+ 492us) server_negotiation.cc:299] Negotiation successful
0621 06:57:26.210944 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":374,"threads_started":1}
I20260621 06:57:26.218806 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.201493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.20.120.254:45907)
0621 06:57:26.201728 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.201737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.202412 (+ 675us) server_negotiation.cc:407] Connection header received
0621 06:57:26.202687 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.202695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.202775 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.202881 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.204088 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.205248 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.207396 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.208094 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.208242 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.211720 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:26.211727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:26.211733 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:26.211794 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:26.215919 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.216637 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.216643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.216653 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.216778 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.217203 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.217215 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.217218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.217525 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:26.217692 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:26.218216 (+ 524us) server_negotiation.cc:299] Negotiation successful
0621 06:57:26.218412 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":81}
I20260621 06:57:26.218832 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.201438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:45907 (local address 127.0.0.1:36630)
0621 06:57:26.202281 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:26.202316 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:26.202520 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:26.203007 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:26.203015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:26.203572 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:26.203845 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:26.203881 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.205479 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.205486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:26.207209 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:26.207223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.208282 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.208290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:26.208440 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.209015 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:26.209040 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:26.211506 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:26.216088 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:26.216100 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:26.216107 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:26.216512 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:26.216890 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:26.216895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:26.216899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:26.217077 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:26.217667 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:26.217674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:26.217867 (+ 193us) client_negotiation.cc:769] Sending connection context
0621 06:57:26.218088 (+ 221us) client_negotiation.cc:240] Negotiation successful
0621 06:57:26.218401 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":518,"threads_started":1}
I20260621 06:57:26.220397 11072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:57:26 dist-test-slave-gmf2 krb5kdc[10844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
Jun 21 06:57:26 dist-test-slave-gmf2 krb5kdc[10844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025046, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.252@KRBTEST.COM for kudu/127.20.120.253@KRBTEST.COM
I20260621 06:57:26.230365 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.215079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.20.120.254:45907)
0621 06:57:26.215667 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.215671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.215690 (+ 19us) server_negotiation.cc:407] Connection header received
0621 06:57:26.215727 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.215730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.215779 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.215893 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.217132 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.218110 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.219513 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.220055 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.220190 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.224668 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:26.224673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:26.224676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:26.224704 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:26.227694 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.228386 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.228392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.228395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.228523 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.228839 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.228844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.228847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.229100 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:26.229357 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:26.229645 (+ 288us) server_negotiation.cc:299] Negotiation successful
0621 06:57:26.229845 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":469,"thread_start_us":273,"threads_started":1}
I20260621 06:57:26.233983 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.221044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:45387 (local address 127.0.0.1:46444)
0621 06:57:26.221247 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:26.221267 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:26.221360 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:26.221631 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:26.221636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:26.222144 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:26.222342 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:26.222357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.223266 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.223274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:26.224195 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:26.224205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.225027 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.225035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:26.225257 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.225680 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:26.225704 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:26.227959 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:26.231623 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:26.231631 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:26.231636 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:26.232119 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:26.232505 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:26.232510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:26.232513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:26.232705 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:26.233309 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:26.233312 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:26.233426 (+ 114us) client_negotiation.cc:769] Sending connection context
0621 06:57:26.233527 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 06:57:26.233702 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260621 06:57:26.234347 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.221212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.20.120.253:45387)
0621 06:57:26.221381 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.221386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.221402 (+ 16us) server_negotiation.cc:407] Connection header received
0621 06:57:26.221447 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.221451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.221503 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.221587 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.222488 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.223126 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.224341 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.224879 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.224969 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.228118 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:26.228123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:26.228126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:26.228153 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:26.231504 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.232269 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.232273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.232275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.232398 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.232831 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.232837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.232840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.233184 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:26.233358 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:26.233665 (+ 307us) server_negotiation.cc:299] Negotiation successful
0621 06:57:26.233882 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 06:57:26.235085 11072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:57:26 dist-test-slave-gmf2 krb5kdc[10844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.253@KRBTEST.COM
Jun 21 06:57:26 dist-test-slave-gmf2 krb5kdc[10844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025046, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.252@KRBTEST.COM for kudu/127.20.120.252@KRBTEST.COM
I20260621 06:57:26.247184 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.231821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.20.120.253:45387)
0621 06:57:26.232370 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.232374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.232393 (+ 19us) server_negotiation.cc:407] Connection header received
0621 06:57:26.232428 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.232431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.232508 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.232612 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.234048 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.235210 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.236548 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.236996 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.237136 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.241586 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:26.241591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:26.241594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:26.241622 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:26.244518 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.245271 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.245274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.245276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.245388 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.245782 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.245788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.245791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.246099 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:26.246345 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:26.246607 (+ 262us) server_negotiation.cc:299] Negotiation successful
0621 06:57:26.246818 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":238,"threads_started":1}
I20260621 06:57:26.247996 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.235752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.20.120.252:36887)
0621 06:57:26.235876 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.235883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.235902 (+ 19us) server_negotiation.cc:407] Connection header received
0621 06:57:26.236070 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.236076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.236126 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.236212 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.236842 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.237479 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.239193 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.239561 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.239650 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.242560 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:26.242565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:26.242567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:26.242592 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:26.245322 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.245886 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.245893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.245896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.246068 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.246392 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.246399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.246402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.246634 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:26.246773 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:26.247113 (+ 340us) server_negotiation.cc:299] Negotiation successful
0621 06:57:26.247637 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 06:57:26.248766 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.235700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58204)
0621 06:57:26.235848 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:26.235864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:26.235951 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:26.236226 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:26.236229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:26.236579 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:26.236703 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:26.236711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.237626 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.237629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:26.239045 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:26.239074 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.239714 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.239720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:26.239921 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.240471 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:26.240487 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:26.242391 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:26.245454 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:26.245459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:26.245463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:26.245778 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:26.246168 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:26.246174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:26.246177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:26.246296 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:26.246740 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:26.246743 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:26.246872 (+ 129us) client_negotiation.cc:769] Sending connection context
0621 06:57:26.248149 (+ 1277us) client_negotiation.cc:240] Negotiation successful
0621 06:57:26.248455 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 06:57:26.252002 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:26.254140 11072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa: Bootstrap starting.
I20260621 06:57:26.254658 10899 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9caed4c752b5471ea799665f2513c75c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36634
I20260621 06:57:26.257189 11072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:26.258251 11072 log.cc:824] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:26.260984 11072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa: No bootstrap required, opened a new log
I20260621 06:57:26.268213 11072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } }
I20260621 06:57:26.268523 11072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae13c8699b94a748e07d54ab3a978aa, State: Initialized, Role: FOLLOWER
I20260621 06:57:26.268940 11072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } }
I20260621 06:57:26.269958 11080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } } }
I20260621 06:57:26.270346 11080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:26.270961 11072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [sys.catalog]: configured and running, proceeding with master startup.
W20260621 06:57:26.280023 11091 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 06:57:26.280160 11091 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 06:57:26.277866 11005 master_runner.cc:186] Error getting master registration for 127.20.120.254:45907: OK, instance_id { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" instance_seqno: 1782025045193293 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 06:57:26.288784 11005 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 06:57:26.333007 10924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } attempt: 4
Jun 21 06:57:26 dist-test-slave-gmf2 krb5kdc[10844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025045, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for kudu/127.20.120.252@KRBTEST.COM
I20260621 06:57:26.347643 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.333753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58212)
0621 06:57:26.333962 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:26.333983 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:26.334116 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:26.334489 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:26.334495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:26.334912 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:26.335070 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:26.335079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.336068 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.336076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:26.336970 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:26.336982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.337687 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.337693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:26.337949 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.338597 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:26.338614 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:26.340841 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:26.344891 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:26.344896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:26.344902 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:26.345382 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:26.345789 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:26.345794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:26.345797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:26.345983 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:26.346682 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:26.346686 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:26.346836 (+ 150us) client_negotiation.cc:769] Sending connection context
0621 06:57:26.346969 (+ 133us) client_negotiation.cc:240] Negotiation successful
0621 06:57:26.347239 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 06:57:26.347702 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.333868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.20.120.252:36887)
0621 06:57:26.334087 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.334092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.334116 (+ 24us) server_negotiation.cc:407] Connection header received
0621 06:57:26.334274 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.334280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.334351 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.334454 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.335225 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.335938 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.337116 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.337541 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.337667 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.341037 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:26.341044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:26.341050 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:26.341087 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:26.344747 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.345510 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.345518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.345526 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.345677 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.346111 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.346116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.346119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.346558 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:26.346710 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:26.347127 (+ 417us) server_negotiation.cc:299] Negotiation successful
0621 06:57:26.347339 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 06:57:26.353564 10924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c: Bootstrap starting.
I20260621 06:57:26.357358 10924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:26.358564 10924 log.cc:824] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:26.361609 10924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c: No bootstrap required, opened a new log
I20260621 06:57:26.368795 10924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } }
I20260621 06:57:26.369186 10924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9caed4c752b5471ea799665f2513c75c, State: Initialized, Role: FOLLOWER
I20260621 06:57:26.369664 10924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } }
I20260621 06:57:26.370882 11094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } } }
I20260621 06:57:26.371434 11094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:26.372006 10924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:26.372071 10899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 06:57:26.372249 10899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:26.376188 10899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } }
I20260621 06:57:26.378176 10899 leader_election.cc:288] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [CANDIDATE]: Term 1 election: Requested vote from peers 8cf5361404f94642a6d48eac5dcd2b71 (127.20.120.253:45387), 5ae13c8699b94a748e07d54ab3a978aa (127.20.120.252:36887)
I20260621 06:57:26.379256 11047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9caed4c752b5471ea799665f2513c75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5ae13c8699b94a748e07d54ab3a978aa"
I20260621 06:57:26.379748 11047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:26.383224 11047 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9caed4c752b5471ea799665f2513c75c in term 1.
I20260621 06:57:26.383941 10874 leader_election.cc:302] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ae13c8699b94a748e07d54ab3a978aa, 9caed4c752b5471ea799665f2513c75c; no voters:
I20260621 06:57:26.385007 11094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:26.386133 11094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [term 1 LEADER]: Becoming Leader. State: Replica: 9caed4c752b5471ea799665f2513c75c, State: Running, Role: LEADER
I20260621 06:57:26.386802 11094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } }
I20260621 06:57:26.393767 11102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9caed4c752b5471ea799665f2513c75c. Latest consensus state: current_term: 1 leader_uuid: "9caed4c752b5471ea799665f2513c75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } } }
I20260621 06:57:26.394164 11102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:26.406751 11106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260621 06:57:26.418354 11026 tablet.cc:2369] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:57:26.420955 11106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 06:57:26.421443 10998 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } attempt: 5
W20260621 06:57:26.432705 10878 tablet.cc:2369] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:57:26.435570 10857 master_runner.cc:186] Error getting master registration for 127.20.120.253:45387: OK, instance_id { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" instance_seqno: 1782025045694048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 06:57:26.478820 10857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 06:57:26.490880 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.411359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58218 (local address 127.20.120.252:36887)
0621 06:57:26.411597 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.411606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.419685 (+ 8079us) server_negotiation.cc:407] Connection header received
0621 06:57:26.419967 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.419976 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.420089 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.420212 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.425125 (+ 4913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.426861 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.457538 (+ 30677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.458559 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.458697 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.482134 (+ 23437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:26.482143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:26.482150 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:26.482202 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:26.485893 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.487164 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.487169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.487172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.487340 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.488070 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.488076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.488080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.488496 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:26.488794 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:26.489640 (+ 846us) server_negotiation.cc:299] Negotiation successful
0621 06:57:26.490094 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":80}
I20260621 06:57:26.494119 11047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [term 1 FOLLOWER]: Refusing update from remote peer 9caed4c752b5471ea799665f2513c75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:57:26.495172 11102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Jun 21 06:57:26 dist-test-slave-gmf2 krb5kdc[10844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025045, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.253@KRBTEST.COM for kudu/127.20.120.252@KRBTEST.COM
I20260621 06:57:26.519904 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.418331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.20.120.254:45907)
0621 06:57:26.419322 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.419330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.423750 (+ 4420us) server_negotiation.cc:407] Connection header received
0621 06:57:26.429836 (+ 6086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.429846 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.429974 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.430079 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.461528 (+ 31449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.463027 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.480217 (+ 17190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.481037 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.481161 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.499942 (+ 18781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:26.499950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:26.499955 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:26.499997 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:26.504220 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.511530 (+ 7310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.511540 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.511546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.511728 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.512266 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.512275 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.512280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.512778 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:26.513078 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:26.519084 (+ 6006us) server_negotiation.cc:299] Negotiation successful
0621 06:57:26.519401 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814}
I20260621 06:57:26.527056 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.424332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.20.120.253:45387)
0621 06:57:26.424609 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.424619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.453935 (+ 29316us) server_negotiation.cc:407] Connection header received
0621 06:57:26.454521 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.454532 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.454646 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.454737 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.472080 (+ 17343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.473555 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.486491 (+ 12936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.487181 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.487284 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.513241 (+ 25957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:26.513250 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:26.513257 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:26.513302 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:26.517489 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.522534 (+ 5045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.522541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.522547 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.522720 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.524871 (+ 2151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.524878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.524885 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.525329 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:26.525623 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:26.526274 (+ 651us) server_negotiation.cc:299] Negotiation successful
0621 06:57:26.526589 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260621 06:57:26.541291 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.422143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:36887 (local address 127.0.0.1:58234)
0621 06:57:26.422474 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:26.422499 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:26.422633 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:26.479931 (+ 57298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:26.479939 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:26.480515 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:26.480767 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:26.480783 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.495664 (+ 14881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.495671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:26.497133 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:26.497149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.505325 (+ 8176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.505333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:26.505682 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.506451 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:26.506482 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:26.509791 (+ 3309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:26.533729 (+ 23938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:26.533733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:26.533737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:26.534219 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:26.537877 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:26.537882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:26.537886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:26.538159 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:26.540069 (+ 1910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:26.540077 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:26.540262 (+ 185us) client_negotiation.cc:769] Sending connection context
0621 06:57:26.540404 (+ 142us) client_negotiation.cc:240] Negotiation successful
0621 06:57:26.540792 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260621 06:57:26.543047 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:26.422938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.20.120.252:36887)
0621 06:57:26.479544 (+ 56606us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.479550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.479578 (+ 28us) server_negotiation.cc:407] Connection header received
0621 06:57:26.479645 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.479650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.479737 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.479858 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.488242 (+ 8384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.489848 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.499047 (+ 9199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:26.499718 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:26.510022 (+ 10304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:26.510367 (+ 345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:26.510376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:26.510383 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:26.510440 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:26.531765 (+ 21325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.534423 (+ 2658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.534430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.534438 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.534640 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:26.538320 (+ 3680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:26.538326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:26.538330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:26.538790 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:26.542009 (+ 3219us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:26.542167 (+ 158us) server_negotiation.cc:299] Negotiation successful
0621 06:57:26.542472 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":301,"server-negotiator.queue_time_us":56437,"thread_start_us":56154,"threads_started":1}
I20260621 06:57:26.544332 11080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9caed4c752b5471ea799665f2513c75c. Latest consensus state: current_term: 1 leader_uuid: "9caed4c752b5471ea799665f2513c75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } } }
I20260621 06:57:26.544632 11080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:26.548986 10998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71: Bootstrap starting.
I20260621 06:57:26.555756 10998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:26.557039 10998 log.cc:824] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:26.561324 11118 mvcc.cc:205] Tried to move back new op lower bound from 7299174590336507904 to 7299174590013480960. Current Snapshot: MvccSnapshot[applied={T|T < 7299174590336507904}]
I20260621 06:57:26.563489 11119 mvcc.cc:205] Tried to move back new op lower bound from 7299174590336507904 to 7299174590013480960. Current Snapshot: MvccSnapshot[applied={T|T < 7299174590336507904}]
I20260621 06:57:26.567744 11080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9caed4c752b5471ea799665f2513c75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } } }
I20260621 06:57:26.568080 11080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:26.573827 10998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71: No bootstrap required, opened a new log
I20260621 06:57:26.583464 11094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9caed4c752b5471ea799665f2513c75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } } }
I20260621 06:57:26.583783 11094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:26.584173 11106 catalog_manager.cc:1347] Generated new cluster ID: 8ebb71cad8e34469a55f97c52873f05e
I20260621 06:57:26.584303 11106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:26.591212 10998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } }
I20260621 06:57:26.591805 10998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf5361404f94642a6d48eac5dcd2b71, State: Initialized, Role: FOLLOWER
I20260621 06:57:26.592367 10998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } }
I20260621 06:57:26.600819 11124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } } }
I20260621 06:57:26.603210 11124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:26.603084 10998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:26.605233 10973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9caed4c752b5471ea799665f2513c75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8cf5361404f94642a6d48eac5dcd2b71"
I20260621 06:57:26.605829 10973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:26.611572 10973 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9caed4c752b5471ea799665f2513c75c in term 1.
I20260621 06:57:26.612484 10972 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [term 1 FOLLOWER]: Refusing update from remote peer 9caed4c752b5471ea799665f2513c75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:57:26.614814 11102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:57:26.659389 11124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9caed4c752b5471ea799665f2513c75c. Latest consensus state: current_term: 1 leader_uuid: "9caed4c752b5471ea799665f2513c75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } } }
I20260621 06:57:26.659721 11124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:26.660177 11131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9caed4c752b5471ea799665f2513c75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } } }
I20260621 06:57:26.661394 11131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:26.681453 11102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9caed4c752b5471ea799665f2513c75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9caed4c752b5471ea799665f2513c75c" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 45907 } } peers { permanent_uuid: "8cf5361404f94642a6d48eac5dcd2b71" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 45387 } } peers { permanent_uuid: "5ae13c8699b94a748e07d54ab3a978aa" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 36887 } } }
I20260621 06:57:26.681742 11102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:26.695683 11106 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:26.698176 11106 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:26.737015 11139 catalog_manager.cc:1259] Loaded cluster ID: 8ebb71cad8e34469a55f97c52873f05e
I20260621 06:57:26.737121 11139 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71: loading cluster ID for follower catalog manager: success
I20260621 06:57:26.740932 11139 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71: acquiring CA information for follower catalog manager: success
W20260621 06:57:26.742267 11139 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260621 06:57:26.760897 11106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9caed4c752b5471ea799665f2513c75c: Generated new TSK 0
I20260621 06:57:26.762172 11106 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 06:57:26.867897 11116 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:26.855164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.20.120.252:36887)
0621 06:57:26.855374 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.855383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.865591 (+ 10208us) server_negotiation.cc:407] Connection header received
0621 06:57:26.866237 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.866244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.866337 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.866426 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.867649 (+ 1223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58250: BlockingRecv error: recv got EOF from 127.0.0.1:58250 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260621 06:57:26.870476 11077 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:26.855136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.20.120.254:45907)
0621 06:57:26.855323 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.855330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.868719 (+ 13389us) server_negotiation.cc:407] Connection header received
0621 06:57:26.869018 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.869025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.869156 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.869231 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.870220 (+ 989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36654: BlockingRecv error: recv got EOF from 127.0.0.1:36654 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260621 06:57:26.873384 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:26.871008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.20.120.253:45387)
0621 06:57:26.871200 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.871208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.871392 (+ 184us) server_negotiation.cc:407] Connection header received
0621 06:57:26.871700 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.871705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.871854 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.871980 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.873077 (+ 1097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46478: BlockingRecv error: recv got EOF from 127.0.0.1:46478 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260621 06:57:26.886292 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:26.877949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.20.120.253:45387)
0621 06:57:26.878109 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.878115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.884443 (+ 6328us) server_negotiation.cc:407] Connection header received
0621 06:57:26.884735 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.884740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.884885 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.884986 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.886074 (+ 1088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46486: BlockingRecv error: recv got EOF from 127.0.0.1:46486 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260621 06:57:26.895969 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:26.891615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.20.120.253:45387)
0621 06:57:26.891779 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.891785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.894435 (+ 2650us) server_negotiation.cc:407] Connection header received
0621 06:57:26.894629 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.894635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.894724 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.894837 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.895669 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46496: BlockingRecv error: recv got EOF from 127.0.0.1:46496 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260621 06:57:26.906226 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:26.903864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.20.120.253:45387)
0621 06:57:26.904054 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.904061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.904632 (+ 571us) server_negotiation.cc:407] Connection header received
0621 06:57:26.904872 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.904878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.904994 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.905116 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.905977 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46500: BlockingRecv error: recv got EOF from 127.0.0.1:46500 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260621 06:57:26.915189 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:26.913000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.20.120.253:45387)
0621 06:57:26.913206 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.913212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.913695 (+ 483us) server_negotiation.cc:407] Connection header received
0621 06:57:26.913863 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.913868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.914009 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.914114 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.914921 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46512: BlockingRecv error: recv got EOF from 127.0.0.1:46512 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260621 06:57:26.928269 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:26.926147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.20.120.253:45387)
0621 06:57:26.926322 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.926330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.926837 (+ 507us) server_negotiation.cc:407] Connection header received
0621 06:57:26.927017 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.927024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.927138 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.927258 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.928059 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46518: BlockingRecv error: recv got EOF from 127.0.0.1:46518 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260621 06:57:26.949841 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:26.947570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.20.120.253:45387)
0621 06:57:26.947766 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.947773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.948290 (+ 517us) server_negotiation.cc:407] Connection header received
0621 06:57:26.948494 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.948499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.948594 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.948709 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.949548 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46524: BlockingRecv error: recv got EOF from 127.0.0.1:46524 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260621 06:57:26.991463 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:26.989239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.20.120.253:45387)
0621 06:57:26.989445 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:26.989452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:26.989946 (+ 494us) server_negotiation.cc:407] Connection header received
0621 06:57:26.990163 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:26.990170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:26.990280 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:26.990397 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:26.991225 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46536: BlockingRecv error: recv got EOF from 127.0.0.1:46536 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260621 06:57:27.061097 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:27.059068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46552 (local address 127.20.120.253:45387)
0621 06:57:27.059283 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:27.059289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:27.059713 (+ 424us) server_negotiation.cc:407] Connection header received
0621 06:57:27.059853 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:27.059859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:27.059972 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:27.060102 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:27.060864 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46552: BlockingRecv error: recv got EOF from 127.0.0.1:46552 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260621 06:57:27.194437 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:27.192417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.20.120.253:45387)
0621 06:57:27.192650 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:27.192656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:27.193022 (+ 366us) server_negotiation.cc:407] Connection header received
0621 06:57:27.193220 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:27.193224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:27.193292 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:27.193376 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:27.194221 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46554: BlockingRecv error: recv got EOF from 127.0.0.1:46554 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:57:27.282084 11091 catalog_manager.cc:1259] Loaded cluster ID: 8ebb71cad8e34469a55f97c52873f05e
I20260621 06:57:27.282168 11091 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa: loading cluster ID for follower catalog manager: success
I20260621 06:57:27.284507 11091 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa: acquiring CA information for follower catalog manager: success
I20260621 06:57:27.286221 11091 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ae13c8699b94a748e07d54ab3a978aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 06:57:27.456204 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:27.454208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.20.120.253:45387)
0621 06:57:27.454439 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:27.454447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:27.454687 (+ 240us) server_negotiation.cc:407] Connection header received
0621 06:57:27.454912 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:27.454922 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:27.454993 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:27.455126 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:27.455903 (+ 777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46558: BlockingRecv error: recv got EOF from 127.0.0.1:46558 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 06:57:27.744889 11139 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8cf5361404f94642a6d48eac5dcd2b71: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 06:57:27.977375 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:27.971752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.20.120.253:45387)
0621 06:57:27.972304 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:27.972308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:27.972402 (+ 94us) server_negotiation.cc:407] Connection header received
0621 06:57:27.972568 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:27.972572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:27.972714 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:27.972917 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:57:27.973269 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:27.974019 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:27.975383 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:27.975756 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:27.975853 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:27.976276 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:57:27.976781 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:57:27.976923 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:27.976997 (+ 74us) server_negotiation.cc:299] Negotiation successful
0621 06:57:27.977058 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":265,"threads_started":1}
I20260621 06:57:27.987950 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:27.983007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58264 (local address 127.20.120.252:36887)
0621 06:57:27.983406 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:27.983410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:27.983427 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:57:27.983615 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:27.983619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:27.983749 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:27.983884 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:57:27.984199 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:27.984869 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:27.986126 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:27.986496 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:27.986585 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:27.986987 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:57:27.987459 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:57:27.987575 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:27.987663 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 06:57:27.987719 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":224,"threads_started":1}
I20260621 06:57:28.001518 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:27.994265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46582 (local address 127.20.120.253:45387)
0621 06:57:27.994503 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:27.994511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:27.995827 (+ 1316us) server_negotiation.cc:407] Connection header received
0621 06:57:27.996088 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:27.996093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:27.996168 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:27.996226 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:57:27.996899 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:27.997767 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:27.999453 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:28.000110 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:28.000247 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:28.000720 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:57:28.000860 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:57:28.000969 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:28.001148 (+ 179us) server_negotiation.cc:299] Negotiation successful
0621 06:57:28.001223 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 06:57:28.001616 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:27.993583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.20.120.252:36887)
0621 06:57:27.994909 (+ 1326us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:27.994918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:27.995585 (+ 667us) server_negotiation.cc:407] Connection header received
0621 06:57:27.995847 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:27.995852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:27.995921 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:27.996005 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:57:27.996450 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:27.997323 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:27.999563 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:28.000184 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:28.000289 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:28.000844 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:57:28.000978 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:57:28.001063 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:28.001249 (+ 186us) server_negotiation.cc:299] Negotiation successful
0621 06:57:28.001323 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174}
I20260621 06:57:28.002741 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:27.993412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.20.120.254:45907)
0621 06:57:27.994949 (+ 1537us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:27.994958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:27.994981 (+ 23us) server_negotiation.cc:407] Connection header received
0621 06:57:27.995116 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:27.995124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:27.995341 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:27.996090 (+ 749us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:57:27.996258 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:27.997204 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:27.999521 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:27.999928 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:28.000057 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:28.000677 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:57:28.001062 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:57:28.001515 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:28.001564 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 06:57:28.002494 (+ 930us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1352,"thread_start_us":1245,"threads_started":1}
I20260621 06:57:28.007195 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10857
I20260621 06:57:28.008288 10919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:57:28.119167 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 10857
I20260621 06:57:28.140501 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 10931
I20260621 06:57:28.141642 10993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:57:28.243961 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 10931
I20260621 06:57:28.263466 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 11005
I20260621 06:57:28.264529 11067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:57:28.367784 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 11005
2026-06-21T06:57:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4928 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 06:57:28 dist-test-slave-gmf2 krb5kdc[11224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 06:57:28 dist-test-slave-gmf2 krb5kdc[11224](info): set up 2 sockets
Jun 21 06:57:28 dist-test-slave-gmf2 krb5kdc[11224](info): commencing operation
krb5kdc: starting...
W20260621 06:57:29.462301 20963 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/test-user.keytab.
Jun 21 06:57:29 dist-test-slave-gmf2 krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T06:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T06:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.120.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
Entry for principal kudu/127.20.120.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab.
I20260621 06:57:29.693112 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:37429
--webserver_interface=127.20.120.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:37429,127.20.120.253:44405,127.20.120.252:33597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 06:57:30.007952 11237 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 06:57:30.008301 11237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:30.008410 11237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:30.008548 11237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:30.015866 11237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:30.015962 11237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:30.016047 11237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:30.016139 11237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:30.016247 11237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:30.026262 11237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:37429,127.20.120.253:44405,127.20.120.252:33597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.254.keytab
--principal=kudu/127.20.120.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.254:37429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:30.027695 11237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:30.029044 11237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:30.040472 11245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:30.040591 11243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:30.040558 11242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:30.041388 11237 server_base.cc:1034] running on GCE node
I20260621 06:57:30.042236 11237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:30.043577 11237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:30.044826 11237 hybrid_clock.cc:648] HybridClock initialized: now 1782025050044764 us; error 38 us; skew 500 ppm
I20260621 06:57:30.045167 11237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:30 dist-test-slave-gmf2 krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025050, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:30.048226 11237 init.cc:377] Logged in from keytab as kudu/127.20.120.254@KRBTEST.COM (short username kudu)
I20260621 06:57:30.049616 11237 webserver.cc:460] Webserver started at http://127.20.120.254:39651/ using document root <none> and password file <none>
I20260621 06:57:30.050365 11237 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:30.050511 11237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:30.050825 11237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:30.053718 11237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/data/instance:
uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419"
format_stamp: "Formatted at 2026-06-21 06:57:30 on dist-test-slave-gmf2"
server_key: "e618b6bad7e7132fce12fda330569be9"
server_key_iv: "d8280b68632ab39d50b24401dd4b1e25"
server_key_version: "encryptionkey@0"
I20260621 06:57:30.054649 11237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/wal/instance:
uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419"
format_stamp: "Formatted at 2026-06-21 06:57:30 on dist-test-slave-gmf2"
server_key: "e618b6bad7e7132fce12fda330569be9"
server_key_iv: "d8280b68632ab39d50b24401dd4b1e25"
server_key_version: "encryptionkey@0"
I20260621 06:57:30.059568 11237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 06:57:30.062932 11252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:30.064218 11237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 06:57:30.064401 11237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/wal
uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419"
format_stamp: "Formatted at 2026-06-21 06:57:30 on dist-test-slave-gmf2"
server_key: "e618b6bad7e7132fce12fda330569be9"
server_key_iv: "d8280b68632ab39d50b24401dd4b1e25"
server_key_version: "encryptionkey@0"
I20260621 06:57:30.064671 11237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:30.081275 11237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:30.085878 11237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:30.086163 11237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:30.103943 11237 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.254:37429
I20260621 06:57:30.103960 11303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.254:37429 every 8 connection(s)
I20260621 06:57:30.105357 11237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/data/info.pb
I20260621 06:57:30.109230 11304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:30.115684 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 11237
I20260621 06:57:30.115873 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-0/wal/instance
I20260621 06:57:30.115257 11304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } has no permanent_uuid. Determining permanent_uuid...
I20260621 06:57:30.116304 20963 external_mini_cluster.cc:1428] Setting key cc329c90fdcd3905e438d7891a7cb1c3
Jun 21 06:57:30 dist-test-slave-gmf2 krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025050, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:57:30.139186 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.118379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.20.120.254:37429)
0621 06:57:30.119200 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:30.119216 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:30.119752 (+ 536us) server_negotiation.cc:407] Connection header received
0621 06:57:30.121665 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:30.121675 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:30.122030 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:30.122237 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:30.123338 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.125063 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.127283 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.127805 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.127925 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:30.132157 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:30.132165 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:30.132177 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:30.132203 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:30.134999 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:30.135864 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:30.135870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:30.135873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:30.136012 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:30.136445 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:30.136448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:30.136450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:30.136941 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:30.137105 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:30.137784 (+ 679us) server_negotiation.cc:299] Negotiation successful
0621 06:57:30.138166 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":582,"thread_start_us":347,"threads_started":1}
I20260621 06:57:30.139225 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.118379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:37429 (local address 127.0.0.1:60042)
0621 06:57:30.119577 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.119617 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:30.121306 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:30.122265 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:30.122270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:30.122798 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:30.123129 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:30.123145 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.125222 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.125229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:30.127148 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:30.127165 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.128047 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.128056 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:30.128260 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:30.128978 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:30.129009 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:30.131935 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:30.135131 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:30.135145 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:30.135160 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:30.135741 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:30.136125 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:30.136132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:30.136139 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:30.136340 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:30.137099 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:30.137110 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:30.137385 (+ 275us) client_negotiation.cc:769] Sending connection context
0621 06:57:30.137655 (+ 270us) client_negotiation.cc:240] Negotiation successful
0621 06:57:30.138356 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":167,"spinlock_wait_cycles":46336,"thread_start_us":762,"threads_started":1}
I20260621 06:57:30.142132 11304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.120.253@KRBTEST.COM; defaulting to no policy
I20260621 06:57:30.143427 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.142891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32828)
0621 06:57:30.143078 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.143241 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260621 06:57:30.143651 11256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.253@KRBTEST.COM" created.
I20260621 06:57:30.145368 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.144972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32840)
0621 06:57:30.145110 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.145196 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260621 06:57:30.145584 11304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:44405: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.253.keytab.
Entry for principal kudu/127.20.120.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.253.keytab.
I20260621 06:57:30.173444 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.253:44405
--webserver_interface=127.20.120.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.253.keytab
--principal=kudu/127.20.120.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:37429,127.20.120.253:44405,127.20.120.252:33597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:57:30.194906 11304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } attempt: 1
I20260621 06:57:30.196138 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.195693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32846)
0621 06:57:30.195896 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.196010 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260621 06:57:30.197104 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.196756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32858)
0621 06:57:30.196899 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.196973 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 06:57:30.197301 11304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:44405: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
I20260621 06:57:30.265635 11304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } attempt: 2
I20260621 06:57:30.266906 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.266400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32864)
0621 06:57:30.266676 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.266774 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260621 06:57:30.267926 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.267587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32880)
0621 06:57:30.267720 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.267786 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 06:57:30.268194 11304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:44405: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
I20260621 06:57:30.359542 11304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } attempt: 3
I20260621 06:57:30.360824 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.360345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32892)
0621 06:57:30.360600 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.360694 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260621 06:57:30.361874 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.361508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32896)
0621 06:57:30.361669 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.361749 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 06:57:30.362169 11304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:44405: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
W20260621 06:57:30.499311 11311 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 06:57:30.499722 11311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:30.499846 11311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:30.499975 11311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260621 06:57:30.505405 11304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } attempt: 4
I20260621 06:57:30.506717 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.506213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32904)
0621 06:57:30.506496 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.506583 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260621 06:57:30.507658 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.507328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32912)
0621 06:57:30.507458 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.507549 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260621 06:57:30.507840 11304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.253:44405: Network error: Client connection negotiation failed: client connection to 127.20.120.253:44405: connect: Connection refused (error 111)
W20260621 06:57:30.508050 11311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:30.508152 11311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:30.508239 11311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:30.508316 11311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:30.508399 11311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:30.518340 11311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:37429,127.20.120.253:44405,127.20.120.252:33597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.253.keytab
--principal=kudu/127.20.120.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.253:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.120.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:30.519671 11311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:30.521036 11311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:30.532902 11316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:30.532930 11317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:30.533073 11319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:30.533651 11311 server_base.cc:1034] running on GCE node
I20260621 06:57:30.534485 11311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:30.535884 11311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:30.537098 11311 hybrid_clock.cc:648] HybridClock initialized: now 1782025050537031 us; error 49 us; skew 500 ppm
I20260621 06:57:30.537436 11311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:30 dist-test-slave-gmf2 krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025050, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:30.540606 11311 init.cc:377] Logged in from keytab as kudu/127.20.120.253@KRBTEST.COM (short username kudu)
I20260621 06:57:30.542176 11311 webserver.cc:460] Webserver started at http://127.20.120.253:38103/ using document root <none> and password file <none>
I20260621 06:57:30.542855 11311 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:30.542963 11311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:30.543285 11311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:30.546276 11311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/data/instance:
uuid: "ac8f0dc3e1a144d09e0597a1eb991b38"
format_stamp: "Formatted at 2026-06-21 06:57:30 on dist-test-slave-gmf2"
server_key: "f31c3f5b8cfa3ab44d4ea12adbe4410e"
server_key_iv: "44aa06e5a888c674392450236112f639"
server_key_version: "encryptionkey@0"
I20260621 06:57:30.547091 11311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/wal/instance:
uuid: "ac8f0dc3e1a144d09e0597a1eb991b38"
format_stamp: "Formatted at 2026-06-21 06:57:30 on dist-test-slave-gmf2"
server_key: "f31c3f5b8cfa3ab44d4ea12adbe4410e"
server_key_iv: "44aa06e5a888c674392450236112f639"
server_key_version: "encryptionkey@0"
I20260621 06:57:30.552244 11311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 06:57:30.555735 11326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:30.557067 11311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 06:57:30.557267 11311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/wal
uuid: "ac8f0dc3e1a144d09e0597a1eb991b38"
format_stamp: "Formatted at 2026-06-21 06:57:30 on dist-test-slave-gmf2"
server_key: "f31c3f5b8cfa3ab44d4ea12adbe4410e"
server_key_iv: "44aa06e5a888c674392450236112f639"
server_key_version: "encryptionkey@0"
I20260621 06:57:30.557499 11311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:30.572239 11311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:30.576074 11311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:30.576351 11311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:30.596961 11311 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.253:44405
I20260621 06:57:30.596994 11377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.253:44405 every 8 connection(s)
I20260621 06:57:30.598703 11311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/data/info.pb
I20260621 06:57:30.602463 11378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:30.607688 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 11311
I20260621 06:57:30.607880 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-1/wal/instance
I20260621 06:57:30.608423 20963 external_mini_cluster.cc:1428] Setting key d9361571a6d0109e67648b00f1ce6b24
I20260621 06:57:30.609571 11378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:57:30 dist-test-slave-gmf2 krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025050, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.253@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:57:30.631042 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.613004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.20.120.254:37429)
0621 06:57:30.613216 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:30.613223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:30.614411 (+ 1188us) server_negotiation.cc:407] Connection header received
0621 06:57:30.616294 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:30.616303 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:30.616391 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:30.616515 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:30.617871 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.618621 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.621029 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.621432 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.621539 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:30.625505 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:30.625510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:30.625513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:30.625544 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:30.628064 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:30.628761 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:30.628768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:30.628776 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:30.628895 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:30.629291 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:30.629297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:30.629300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:30.629611 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:30.629764 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:30.630653 (+ 889us) server_negotiation.cc:299] Negotiation successful
0621 06:57:30.630819 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 06:57:30.632649 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.613428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:37429 (local address 127.0.0.1:60050)
0621 06:57:30.614240 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.614284 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:30.616036 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:30.616691 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:30.616698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:30.617187 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:30.617685 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:30.617699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.618781 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.618789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:30.620869 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:30.620886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.621656 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.621665 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:30.622019 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:30.622725 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:30.622762 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:30.625276 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:30.628196 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:30.628205 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:30.628218 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:30.628630 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:30.629007 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:30.629012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:30.629016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:30.629184 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:30.629776 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:30.629787 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:30.630257 (+ 470us) client_negotiation.cc:769] Sending connection context
0621 06:57:30.630590 (+ 333us) client_negotiation.cc:240] Negotiation successful
0621 06:57:30.631165 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":351,"threads_started":1}
WARNING: no policy specified for kudu/127.20.120.252@KRBTEST.COM; defaulting to no policy
I20260621 06:57:30.634394 11378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.120.252@KRBTEST.COM" created.
Jun 21 06:57:30 dist-test-slave-gmf2 krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025050, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.253@KRBTEST.COM for kudu/127.20.120.253@KRBTEST.COM
I20260621 06:57:30.648617 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.635093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32922)
0621 06:57:30.635290 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.635308 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:30.635462 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:30.636469 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:30.636473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:30.636790 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:30.636972 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:30.636980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.638350 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.638355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:30.639188 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:30.639198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.639983 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.639988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:30.640197 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:30.640793 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:30.640808 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:30.643204 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:30.646511 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:30.646515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:30.646517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:30.646843 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:30.647195 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:30.647198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:30.647200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:30.647319 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:30.647999 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:30.648002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:30.648102 (+ 100us) client_negotiation.cc:769] Sending connection context
0621 06:57:30.648212 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 06:57:30.648377 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":40}
I20260621 06:57:30.649078 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.635350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.20.120.253:44405)
0621 06:57:30.636055 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:30.636061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:30.636085 (+ 24us) server_negotiation.cc:407] Connection header received
0621 06:57:30.636124 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:30.636129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:30.636291 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:30.636447 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:30.637146 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.638229 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.639335 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.639838 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.639974 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:30.643402 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:30.643412 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:30.643430 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:30.643456 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:30.646375 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:30.646956 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:30.646966 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:30.646972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:30.647099 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:30.647445 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:30.647452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:30.647459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:30.647883 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:30.648085 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:30.648379 (+ 294us) server_negotiation.cc:299] Negotiation successful
0621 06:57:30.648636 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":561,"thread_start_us":419,"threads_started":1}
I20260621 06:57:30.651175 11378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } has no permanent_uuid. Determining permanent_uuid...
I20260621 06:57:30.652184 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.651776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40744)
0621 06:57:30.651934 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.652019 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260621 06:57:30.652390 11327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
I20260621 06:57:30.653795 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.653495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40750)
0621 06:57:30.653646 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.653708 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 06:57:30.654026 11378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:33597: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.120.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.252.keytab.
Entry for principal kudu/127.20.120.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.252.keytab.
I20260621 06:57:30.667309 20963 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
/tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.252:33597
--webserver_interface=127.20.120.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.252.keytab
--principal=kudu/127.20.120.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.120.212:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.254:37429,127.20.120.253:44405,127.20.120.252:33597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 06:57:30.703285 11378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } attempt: 1
I20260621 06:57:30.704532 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.703961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40766)
0621 06:57:30.704201 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.704331 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260621 06:57:30.705600 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.705202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40770)
0621 06:57:30.705354 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.705449 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 06:57:30.705809 11378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:33597: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
I20260621 06:57:30.774050 11378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } attempt: 2
I20260621 06:57:30.775321 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.774735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40786)
0621 06:57:30.774961 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.775088 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260621 06:57:30.776587 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.776095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40802)
0621 06:57:30.776282 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.776387 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260621 06:57:30.776818 11378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:33597: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
I20260621 06:57:30.807168 11304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } attempt: 5
Jun 21 06:57:30 dist-test-slave-gmf2 krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025050, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for kudu/127.20.120.253@KRBTEST.COM
I20260621 06:57:30.820933 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.808048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32938)
0621 06:57:30.808253 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.808307 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:30.808446 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:30.808818 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:30.808823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:30.809244 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:30.809438 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:30.809447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.810525 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.810532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:30.811405 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:30.811416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.812200 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.812205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:30.812439 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:30.813102 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:30.813121 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:30.815372 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:30.818450 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:30.818457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:30.818478 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:30.818895 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:30.819271 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:30.819276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:30.819280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:30.819421 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:30.819947 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:30.819951 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:30.820106 (+ 155us) client_negotiation.cc:769] Sending connection context
0621 06:57:30.820247 (+ 141us) client_negotiation.cc:240] Negotiation successful
0621 06:57:30.820552 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 06:57:30.820933 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.808231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.20.120.253:44405)
0621 06:57:30.808433 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:30.808439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:30.808461 (+ 22us) server_negotiation.cc:407] Connection header received
0621 06:57:30.808568 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:30.808574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:30.808660 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:30.808773 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:30.809597 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.810388 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.811556 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:30.812026 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:30.812186 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:30.815564 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:30.815569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:30.815572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:30.815604 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:30.818318 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:30.819022 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:30.819034 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:30.819037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:30.819165 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:30.819525 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:30.819532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:30.819536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:30.819819 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:30.819979 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:30.820350 (+ 371us) server_negotiation.cc:299] Negotiation successful
0621 06:57:30.820600 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 06:57:30.822350 11304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } has no permanent_uuid. Determining permanent_uuid...
I20260621 06:57:30.823412 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.823027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40818)
0621 06:57:30.823192 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.823282 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260621 06:57:30.824426 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.824134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40824)
0621 06:57:30.824279 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.824345 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260621 06:57:30.824684 11304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:33597: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
I20260621 06:57:30.868072 11378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } attempt: 3
I20260621 06:57:30.869345 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.868847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40832)
0621 06:57:30.869064 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.869160 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 06:57:30.870525 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.870067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40848)
0621 06:57:30.870330 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.870401 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260621 06:57:30.870733 11378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:33597: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
I20260621 06:57:30.875922 11304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } attempt: 1
I20260621 06:57:30.877060 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.876620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40864)
0621 06:57:30.876811 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.876896 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260621 06:57:30.878134 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.877798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40874)
0621 06:57:30.877967 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.878035 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 06:57:30.878333 11304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:33597: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
I20260621 06:57:30.946576 11304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } attempt: 2
I20260621 06:57:30.947924 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.947367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40876)
0621 06:57:30.947609 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.947717 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260621 06:57:30.949102 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:30.948733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40884)
0621 06:57:30.948915 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:30.948993 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260621 06:57:30.949321 11304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:33597: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
W20260621 06:57:30.999256 11385 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 06:57:30.999643 11385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 06:57:30.999776 11385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 06:57:30.999858 11385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 06:57:31.006951 11385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 06:57:31.007037 11385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 06:57:31.007091 11385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 06:57:31.007139 11385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 06:57:31.007200 11385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 06:57:31.013947 11378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } attempt: 4
I20260621 06:57:31.015075 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.014597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40896)
0621 06:57:31.014826 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:31.014930 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 06:57:31.016098 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.015815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40908)
0621 06:57:31.015954 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:31.016009 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 06:57:31.016326 11378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:33597: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
I20260621 06:57:31.016934 11385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.212:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.254:37429,127.20.120.253:44405,127.20.120.252:33597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/krb5kdc/kudu_127.20.120.252.keytab
--principal=kudu/127.20.120.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.252:33597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.120.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 06:57:31.018177 11385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 06:57:31.019464 11385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 06:57:31.030735 11390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:31.031469 11393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 06:57:31.031476 11391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 06:57:31.032321 11385 server_base.cc:1034] running on GCE node
I20260621 06:57:31.033013 11385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 06:57:31.034309 11385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 06:57:31.035595 11385 hybrid_clock.cc:648] HybridClock initialized: now 1782025051035497 us; error 43 us; skew 500 ppm
I20260621 06:57:31.035907 11385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 06:57:31 dist-test-slave-gmf2 krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025051, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 06:57:31.038949 11385 init.cc:377] Logged in from keytab as kudu/127.20.120.252@KRBTEST.COM (short username kudu)
I20260621 06:57:31.040213 11385 webserver.cc:460] Webserver started at http://127.20.120.252:37061/ using document root <none> and password file <none>
I20260621 06:57:31.040936 11385 fs_manager.cc:359] Metadata directory not provided
I20260621 06:57:31.041051 11385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 06:57:31.041410 11385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 06:57:31.044226 11385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/data/instance:
uuid: "4ce3985d509c407792d39b7cbf5f952e"
format_stamp: "Formatted at 2026-06-21 06:57:31 on dist-test-slave-gmf2"
server_key: "7cf3b10d5b333277e922d02b8d9c485e"
server_key_iv: "ad52aea57fed84f35bde60711359841e"
server_key_version: "encryptionkey@0"
I20260621 06:57:31.045035 11385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/wal/instance:
uuid: "4ce3985d509c407792d39b7cbf5f952e"
format_stamp: "Formatted at 2026-06-21 06:57:31 on dist-test-slave-gmf2"
server_key: "7cf3b10d5b333277e922d02b8d9c485e"
server_key_iv: "ad52aea57fed84f35bde60711359841e"
server_key_version: "encryptionkey@0"
I20260621 06:57:31.050024 11385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260621 06:57:31.053361 11400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 06:57:31.054692 11385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 06:57:31.054859 11385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/data,/tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/wal
uuid: "4ce3985d509c407792d39b7cbf5f952e"
format_stamp: "Formatted at 2026-06-21 06:57:31 on dist-test-slave-gmf2"
server_key: "7cf3b10d5b333277e922d02b8d9c485e"
server_key_iv: "ad52aea57fed84f35bde60711359841e"
server_key_version: "encryptionkey@0"
I20260621 06:57:31.055075 11385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 06:57:31.055524 11304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } attempt: 3
I20260621 06:57:31.056537 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.056096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40924)
0621 06:57:31.056302 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:31.056378 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260621 06:57:31.057588 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.057243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40932)
0621 06:57:31.057429 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:31.057492 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260621 06:57:31.057785 11304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.120.252:33597: Network error: Client connection negotiation failed: client connection to 127.20.120.252:33597: connect: Connection refused (error 111)
I20260621 06:57:31.083523 11385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 06:57:31.087417 11385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 06:57:31.087673 11385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 06:57:31.106178 11385 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.252:33597
I20260621 06:57:31.106200 11451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.120.252:33597 every 8 connection(s)
I20260621 06:57:31.107488 11385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/data/info.pb
I20260621 06:57:31.111728 20963 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu as pid 11385
I20260621 06:57:31.111949 20963 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcf0MiJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782024700637160-20963-0/minicluster-data/master-2/wal/instance
I20260621 06:57:31.112066 11452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 06:57:31.112447 20963 external_mini_cluster.cc:1428] Setting key 56d99b277119185dc308fa01a7b66274
I20260621 06:57:31.121042 11452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:57:31 dist-test-slave-gmf2 krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.252@KRBTEST.COM
Jun 21 06:57:31 dist-test-slave-gmf2 krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025051, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.252@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
I20260621 06:57:31.135360 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.114234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.20.120.252:33597)
0621 06:57:31.115052 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.115063 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.115106 (+ 43us) server_negotiation.cc:407] Connection header received
0621 06:57:31.117021 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.117043 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.117436 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.117834 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.119208 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.120387 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.121657 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.122391 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.122589 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.127073 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:31.127080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:31.127090 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:31.127125 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:31.130923 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.131533 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.131549 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.131558 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.131763 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.132180 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.132183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.132185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.132759 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:31.133036 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:31.133543 (+ 507us) server_negotiation.cc:299] Negotiation successful
0621 06:57:31.133961 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":326,"threads_started":1}
I20260621 06:57:31.139396 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.123590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.254:37429 (local address 127.0.0.1:60066)
0621 06:57:31.124267 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:31.124312 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:31.124509 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:31.124954 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:31.124961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:31.125586 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:31.125860 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:31.125896 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.127257 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.127264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:31.129059 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:31.129072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.129850 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.129856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:31.130038 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.130609 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:31.130631 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:31.133059 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:31.136715 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:31.136723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:31.136729 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:31.137314 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:31.137679 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:31.137682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:31.137684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:31.137866 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:31.138392 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:31.138400 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:31.138599 (+ 199us) client_negotiation.cc:769] Sending connection context
0621 06:57:31.138759 (+ 160us) client_negotiation.cc:240] Negotiation successful
0621 06:57:31.139044 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":353,"threads_started":1}
I20260621 06:57:31.139555 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.123551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60066 (local address 127.20.120.254:37429)
0621 06:57:31.123746 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.123751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.124422 (+ 671us) server_negotiation.cc:407] Connection header received
0621 06:57:31.124678 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.124685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.124768 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.124894 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.126084 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.127097 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.129247 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.129665 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.129761 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.133254 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:31.133261 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:31.133269 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:31.133314 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:31.136577 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.137424 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.137428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.137430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.137554 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.138019 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.138025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.138028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.138262 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:31.138432 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:31.138986 (+ 554us) server_negotiation.cc:299] Negotiation successful
0621 06:57:31.139176 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 06:57:31.140671 11452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:57:31 dist-test-slave-gmf2 krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.254@KRBTEST.COM
Jun 21 06:57:31 dist-test-slave-gmf2 krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025051, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.252@KRBTEST.COM for kudu/127.20.120.253@KRBTEST.COM
I20260621 06:57:31.154635 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.138504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.20.120.254:37429)
0621 06:57:31.139048 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.139051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.139068 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:57:31.139102 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.139105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.139152 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.139260 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.140624 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.141534 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.142803 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.143206 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.143358 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.148158 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:31.148164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:31.148169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:31.148203 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:31.151543 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.152386 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.152394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.152398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.152526 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.152979 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.152984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.152987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.153289 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:31.153537 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:31.154063 (+ 526us) server_negotiation.cc:299] Negotiation successful
0621 06:57:31.154263 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":250,"threads_started":1}
I20260621 06:57:31.154902 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.141209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.253:44405 (local address 127.0.0.1:32954)
0621 06:57:31.141581 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:31.141599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:31.141693 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:31.142102 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:31.142106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:31.142469 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:31.142588 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:31.142595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.143514 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.143521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:31.144494 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:31.144505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.145544 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.145553 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:31.145742 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.146344 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:31.146367 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:31.148677 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:31.152240 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:31.152245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:31.152249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:31.152685 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:31.153108 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:31.153114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:31.153118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:31.153280 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:31.153822 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:31.153827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:31.154043 (+ 216us) client_negotiation.cc:769] Sending connection context
0621 06:57:31.154202 (+ 159us) client_negotiation.cc:240] Negotiation successful
0621 06:57:31.154485 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260621 06:57:31.155611 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.141397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.20.120.253:44405)
0621 06:57:31.141573 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.141579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.141646 (+ 67us) server_negotiation.cc:407] Connection header received
0621 06:57:31.141845 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.141850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.141985 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.142086 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.142724 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.143387 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.144653 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.145232 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.145348 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.148885 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:31.148891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:31.148894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:31.148927 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:31.152084 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.152811 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.152817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.152821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.152981 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.153404 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.153410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.153414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.153688 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:31.153859 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:31.154305 (+ 446us) server_negotiation.cc:299] Negotiation successful
0621 06:57:31.154505 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 06:57:31.156536 11452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 06:57:31 dist-test-slave-gmf2 krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025051, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.252@KRBTEST.COM for kudu/127.20.120.252@KRBTEST.COM
Jun 21 06:57:31 dist-test-slave-gmf2 krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.120.253@KRBTEST.COM
I20260621 06:57:31.172935 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.157395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.20.120.252:33597)
0621 06:57:31.157557 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.157563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.158102 (+ 539us) server_negotiation.cc:407] Connection header received
0621 06:57:31.158251 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.158256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.158325 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.158415 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.159318 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.160329 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.162169 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.162847 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.162981 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.167436 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:31.167440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:31.167443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:31.167469 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:31.170133 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.170933 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.170936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.170938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.171065 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.171473 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.171478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.171481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.171802 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:31.171914 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:31.172357 (+ 443us) server_negotiation.cc:299] Negotiation successful
0621 06:57:31.172559 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":54}
I20260621 06:57:31.173336 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.157148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40936)
0621 06:57:31.157989 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:31.158007 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:31.158140 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:31.158469 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:31.158474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:31.158947 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:31.159161 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:31.159174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.160646 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.160651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:31.162005 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:31.162019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.163117 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.163121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:31.163363 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.163984 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:31.164003 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:31.165965 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:31.170270 (+ 4305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:31.170279 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:31.170283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:31.170806 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:31.171203 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:31.171206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:31.171208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:31.171348 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:31.171924 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:31.171931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:31.172086 (+ 155us) client_negotiation.cc:769] Sending connection context
0621 06:57:31.172241 (+ 155us) client_negotiation.cc:240] Negotiation successful
0621 06:57:31.172871 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":135}
I20260621 06:57:31.174103 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.157216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.20.120.253:44405)
0621 06:57:31.157479 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.157486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.157504 (+ 18us) server_negotiation.cc:407] Connection header received
0621 06:57:31.157553 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.157558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.157627 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.157745 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.159583 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.160312 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.161941 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.162622 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.162774 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.167548 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:31.167556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:31.167565 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:31.167594 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:31.171027 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.171706 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.171710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.171712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.171890 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.172238 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.172243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.172247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.172635 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:31.172928 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:31.173431 (+ 503us) server_negotiation.cc:299] Negotiation successful
0621 06:57:31.173677 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260621 06:57:31.178480 20963 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 06:57:31.179068 11452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e: Bootstrap starting.
I20260621 06:57:31.182588 11452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:31.180899 11279 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60076
I20260621 06:57:31.183740 11452 log.cc:824] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:31.186627 11452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e: No bootstrap required, opened a new log
I20260621 06:57:31.193504 11452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } }
I20260621 06:57:31.193812 11452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce3985d509c407792d39b7cbf5f952e, State: Initialized, Role: FOLLOWER
I20260621 06:57:31.194250 11452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } }
I20260621 06:57:31.195171 11459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } } }
I20260621 06:57:31.195560 11459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:31.196373 11452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [sys.catalog]: configured and running, proceeding with master startup.
W20260621 06:57:31.205617 11470 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 06:57:31.205744 11470 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 06:57:31.203318 11385 master_runner.cc:186] Error getting master registration for 127.20.120.254:37429: OK, instance_id { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" instance_seqno: 1782025050098701 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 06:57:31.212893 11385 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 06:57:31.235037 11304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } attempt: 4
Jun 21 06:57:31 dist-test-slave-gmf2 krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025050, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.254@KRBTEST.COM for kudu/127.20.120.252@KRBTEST.COM
I20260621 06:57:31.251679 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.235729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40944)
0621 06:57:31.236029 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:31.236057 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:31.236236 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:31.236660 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:31.236667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:31.237179 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:31.237405 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:31.237416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.238862 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.238868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:31.240274 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:31.240295 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.241315 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.241322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:31.241540 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.242266 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:31.242299 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:31.244945 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:31.248918 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:31.248924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:31.248928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:31.249435 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:31.249833 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:31.249837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:31.249841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:31.250008 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:31.250644 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:31.250648 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:31.250833 (+ 185us) client_negotiation.cc:769] Sending connection context
0621 06:57:31.250977 (+ 144us) client_negotiation.cc:240] Negotiation successful
0621 06:57:31.251263 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260621 06:57:31.251715 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.235936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.20.120.252:33597)
0621 06:57:31.236145 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.236150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.236171 (+ 21us) server_negotiation.cc:407] Connection header received
0621 06:57:31.236410 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.236416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.236500 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.236617 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.237564 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.238717 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.240427 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.241157 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.241283 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.245127 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:31.245135 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:31.245143 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:31.245186 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:31.248770 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.249553 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.249559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.249562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.249705 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.250110 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.250115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.250119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.250510 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:31.250690 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:31.251123 (+ 433us) server_negotiation.cc:299] Negotiation successful
0621 06:57:31.251352 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 06:57:31.258260 11304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419: Bootstrap starting.
I20260621 06:57:31.262567 11304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:31.263916 11304 log.cc:824] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:31.267387 11304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419: No bootstrap required, opened a new log
I20260621 06:57:31.275142 11304 raft_consensus.cc:348] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } }
I20260621 06:57:31.275573 11304 raft_consensus.cc:729] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58fc7f19dbe49f98d8e27ce2cbaa419, State: Initialized, Role: FOLLOWER
I20260621 06:57:31.276104 11304 consensus_queue.cc:260] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } }
I20260621 06:57:31.277086 11473 sys_catalog.cc:455] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } } }
I20260621 06:57:31.277489 11473 sys_catalog.cc:458] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:31.278448 11304 sys_catalog.cc:564] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:31.278479 11279 raft_consensus.cc:482] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 06:57:31.278676 11279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:31.281423 11279 raft_consensus.cc:504] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } }
I20260621 06:57:31.284070 11279 leader_election.cc:288] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [CANDIDATE]: Term 1 election: Requested vote from peers ac8f0dc3e1a144d09e0597a1eb991b38 (127.20.120.253:44405), 4ce3985d509c407792d39b7cbf5f952e (127.20.120.252:33597)
I20260621 06:57:31.284051 11427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ce3985d509c407792d39b7cbf5f952e"
I20260621 06:57:31.284583 11427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:31.289353 11427 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d58fc7f19dbe49f98d8e27ce2cbaa419 in term 1.
I20260621 06:57:31.290199 11253 leader_election.cc:302] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ce3985d509c407792d39b7cbf5f952e, d58fc7f19dbe49f98d8e27ce2cbaa419; no voters:
I20260621 06:57:31.291005 11473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 06:57:31.301019 11473 raft_consensus.cc:686] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [term 1 LEADER]: Becoming Leader. State: Replica: d58fc7f19dbe49f98d8e27ce2cbaa419, State: Running, Role: LEADER
I20260621 06:57:31.301725 11473 consensus_queue.cc:237] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } }
I20260621 06:57:31.305137 11483 sys_catalog.cc:455] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d58fc7f19dbe49f98d8e27ce2cbaa419. Latest consensus state: current_term: 1 leader_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } } }
I20260621 06:57:31.305583 11483 sys_catalog.cc:458] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:31.314612 11489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 06:57:31.315537 11378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } attempt: 5
Jun 21 06:57:31 dist-test-slave-gmf2 krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025050, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.120.253@KRBTEST.COM for kudu/127.20.120.252@KRBTEST.COM
I20260621 06:57:31.329725 11489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260621 06:57:31.338876 11258 tablet.cc:2369] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 06:57:31.339569 11406 tablet.cc:2369] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 06:57:31.345629 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.316346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.120.252:33597 (local address 127.0.0.1:40952)
0621 06:57:31.316518 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 06:57:31.316539 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 06:57:31.316665 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 06:57:31.317458 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 06:57:31.317463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 06:57:31.318122 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0621 06:57:31.318424 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:31.318442 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.320373 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.320382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:31.322334 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 06:57:31.322354 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.323966 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.323974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 06:57:31.324315 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.325186 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 06:57:31.325219 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 06:57:31.335530 (+ 10311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 06:57:31.341556 (+ 6026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:31.341560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:31.341564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:31.342087 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:31.343254 (+ 1167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 06:57:31.343259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 06:57:31.343262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 06:57:31.343472 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 06:57:31.344526 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 06:57:31.344533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 06:57:31.344713 (+ 180us) client_negotiation.cc:769] Sending connection context
0621 06:57:31.344873 (+ 160us) client_negotiation.cc:240] Negotiation successful
0621 06:57:31.345188 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 06:57:31.346709 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.316897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40952 (local address 127.20.120.252:33597)
0621 06:57:31.317094 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.317102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.317124 (+ 22us) server_negotiation.cc:407] Connection header received
0621 06:57:31.317200 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.317208 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.317299 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.317405 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.318682 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.320145 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.322563 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.323534 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.323692 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.336586 (+ 12894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:31.336592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:31.336596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:31.336636 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:31.341341 (+ 4705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.342219 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.342224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.342228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.342397 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.343610 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.343616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.343620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.343966 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:31.344113 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:31.346071 (+ 1958us) server_negotiation.cc:299] Negotiation successful
0621 06:57:31.346333 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 06:57:31.334517 11237 master_runner.cc:186] Error getting master registration for 127.20.120.253:44405: OK, instance_id { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" instance_seqno: 1782025050589975 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 06:57:31.347437 11237 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 06:57:31.354384 11378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38: Bootstrap starting.
I20260621 06:57:31.356179 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.328440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40964 (local address 127.20.120.252:33597)
0621 06:57:31.329490 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.329501 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.332695 (+ 3194us) server_negotiation.cc:407] Connection header received
0621 06:57:31.333097 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.333105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.333188 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.333265 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.337372 (+ 4107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.338706 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.341728 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.342413 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.342499 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.348386 (+ 5887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:31.348393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:31.348399 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:31.348434 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:31.352050 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.353289 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.353294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.353297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.353432 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.354087 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.354091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.354094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.354436 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:31.354690 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:31.355505 (+ 815us) server_negotiation.cc:299] Negotiation successful
0621 06:57:31.355728 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":431,"threads_started":1}
I20260621 06:57:31.358937 11378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38: Neither blocks nor log segments found. Creating new log.
I20260621 06:57:31.362094 11378 log.cc:824] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38: Log is configured to *not* fsync() on all Append() calls
I20260621 06:57:31.367426 11427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [term 1 FOLLOWER]: Refusing update from remote peer d58fc7f19dbe49f98d8e27ce2cbaa419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:57:31.369359 11483 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:57:31.370841 11378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38: No bootstrap required, opened a new log
I20260621 06:57:31.382052 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.331684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.20.120.254:37429)
0621 06:57:31.331857 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.331863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.348287 (+ 16424us) server_negotiation.cc:407] Connection header received
0621 06:57:31.349202 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.349207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.349306 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.349375 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.351134 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.352288 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.358043 (+ 5755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.358811 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.358922 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.374073 (+ 15151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:31.374080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:31.374101 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:31.374142 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:31.377765 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.378662 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.378668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.378672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.378819 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.379304 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.379309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.379312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.379670 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:31.379908 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:31.381392 (+ 1484us) server_negotiation.cc:299] Negotiation successful
0621 06:57:31.381637 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 06:57:31.383312 11378 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } }
I20260621 06:57:31.383793 11378 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8f0dc3e1a144d09e0597a1eb991b38, State: Initialized, Role: FOLLOWER
I20260621 06:57:31.384351 11378 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } }
I20260621 06:57:31.391458 11500 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } } }
I20260621 06:57:31.391827 11500 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:31.393644 11378 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 06:57:31.394018 11352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [term 0 FOLLOWER]: Advancing to term 1
I20260621 06:57:31.398029 11353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38"
I20260621 06:57:31.399526 11352 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [term 1 FOLLOWER]: Refusing update from remote peer d58fc7f19dbe49f98d8e27ce2cbaa419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 06:57:31.403899 11353 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d58fc7f19dbe49f98d8e27ce2cbaa419 in term 1.
I20260621 06:57:31.406900 11483 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 06:57:31.422533 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:31.334506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.20.120.253:44405)
0621 06:57:31.334675 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.334681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.357541 (+ 22860us) server_negotiation.cc:407] Connection header received
0621 06:57:31.357935 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.357940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.358013 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.358082 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.361513 (+ 3431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.362746 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.376958 (+ 14212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:31.377619 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:31.377734 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:31.412192 (+ 34458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 06:57:31.412202 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 06:57:31.412210 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 06:57:31.412259 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 06:57:31.416949 (+ 4690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.418678 (+ 1729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.418687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.418693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.418922 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 06:57:31.419844 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 06:57:31.419852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 06:57:31.419858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 06:57:31.420287 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 06:57:31.420582 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:31.421666 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0621 06:57:31.422031 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 06:57:31.427346 11459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d58fc7f19dbe49f98d8e27ce2cbaa419. Latest consensus state: current_term: 1 leader_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } } }
I20260621 06:57:31.427704 11459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:31.432310 11497 mvcc.cc:205] Tried to move back new op lower bound from 7299174610389843968 to 7299174610137509888. Current Snapshot: MvccSnapshot[applied={T|T < 7299174610389843968}]
I20260621 06:57:31.441835 11500 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d58fc7f19dbe49f98d8e27ce2cbaa419. Latest consensus state: current_term: 1 leader_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } } }
I20260621 06:57:31.442194 11500 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:31.443274 11499 mvcc.cc:205] Tried to move back new op lower bound from 7299174610389843968 to 7299174610137509888. Current Snapshot: MvccSnapshot[applied={T|T < 7299174610389843968}]
I20260621 06:57:31.460202 11473 sys_catalog.cc:455] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } } }
I20260621 06:57:31.460636 11473 sys_catalog.cc:458] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:31.460965 11473 sys_catalog.cc:455] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } } }
I20260621 06:57:31.461265 11473 sys_catalog.cc:458] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419 [sys.catalog]: This master's current role is: LEADER
I20260621 06:57:31.461339 11500 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } } }
I20260621 06:57:31.461694 11500 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:31.466753 11489 catalog_manager.cc:1347] Generated new cluster ID: 970f9673d4ca4d149dd80b6dffbda833
I20260621 06:57:31.466884 11489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 06:57:31.468096 11459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d58fc7f19dbe49f98d8e27ce2cbaa419" member_type: VOTER last_known_addr { host: "127.20.120.254" port: 37429 } } peers { permanent_uuid: "ac8f0dc3e1a144d09e0597a1eb991b38" member_type: VOTER last_known_addr { host: "127.20.120.253" port: 44405 } } peers { permanent_uuid: "4ce3985d509c407792d39b7cbf5f952e" member_type: VOTER last_known_addr { host: "127.20.120.252" port: 33597 } } }
I20260621 06:57:31.468405 11459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e [sys.catalog]: This master's current role is: FOLLOWER
I20260621 06:57:31.486848 11518 catalog_manager.cc:1259] Loaded cluster ID: 970f9673d4ca4d149dd80b6dffbda833
I20260621 06:57:31.487038 11518 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38: loading cluster ID for follower catalog manager: success
W20260621 06:57:31.488492 11518 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260621 06:57:31.489107 11518 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260621 06:57:31.531157 11489 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 06:57:31.535806 11489 catalog_manager.cc:1504] Loading token signing keys...
I20260621 06:57:31.550971 11489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d58fc7f19dbe49f98d8e27ce2cbaa419: Generated new TSK 0
I20260621 06:57:31.552218 11489 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 06:57:31.680617 11493 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:31.666618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.20.120.252:33597)
0621 06:57:31.666828 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.666834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.678395 (+ 11561us) server_negotiation.cc:407] Connection header received
0621 06:57:31.678714 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.678721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.678816 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.678902 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.680309 (+ 1407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40976: BlockingRecv error: recv got EOF from 127.0.0.1:40976 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260621 06:57:31.683300 11457 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:31.675174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.20.120.254:37429)
0621 06:57:31.675362 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.675369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.681621 (+ 6252us) server_negotiation.cc:407] Connection header received
0621 06:57:31.681849 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.681855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.681986 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.682062 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.683029 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60096: BlockingRecv error: recv got EOF from 127.0.0.1:60096 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260621 06:57:31.686331 11382 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:31.684291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.20.120.253:44405)
0621 06:57:31.684497 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.684503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.684719 (+ 216us) server_negotiation.cc:407] Connection header received
0621 06:57:31.684964 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.684972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.685077 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.685152 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.686093 (+ 941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32990: BlockingRecv error: recv got EOF from 127.0.0.1:32990 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260621 06:57:31.729127 11382 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:31.719852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.20.120.253:44405)
0621 06:57:31.720057 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.720064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.726963 (+ 6899us) server_negotiation.cc:407] Connection header received
0621 06:57:31.727513 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.727520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.727652 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.727734 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.728818 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33006: BlockingRecv error: recv got EOF from 127.0.0.1:33006 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260621 06:57:31.772367 11382 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:31.763066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.20.120.253:44405)
0621 06:57:31.763293 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.763300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.770538 (+ 7238us) server_negotiation.cc:407] Connection header received
0621 06:57:31.770854 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.770861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.770979 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.771060 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.772123 (+ 1063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33022: BlockingRecv error: recv got EOF from 127.0.0.1:33022 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260621 06:57:31.817945 11382 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:31.808598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.20.120.253:44405)
0621 06:57:31.808800 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.808807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.815830 (+ 7023us) server_negotiation.cc:407] Connection header received
0621 06:57:31.816391 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.816397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.816522 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.816605 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.817648 (+ 1043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33030: BlockingRecv error: recv got EOF from 127.0.0.1:33030 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260621 06:57:31.872500 11382 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:31.859299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.20.120.253:44405)
0621 06:57:31.859590 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.859598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.870641 (+ 11043us) server_negotiation.cc:407] Connection header received
0621 06:57:31.870911 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.870920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.871027 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.871106 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.872215 (+ 1109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33036: BlockingRecv error: recv got EOF from 127.0.0.1:33036 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
W20260621 06:57:31.924437 11382 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:31.919757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.20.120.253:44405)
0621 06:57:31.919946 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.919953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.922632 (+ 2679us) server_negotiation.cc:407] Connection header received
0621 06:57:31.922875 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.922882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.922998 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.923123 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.924160 (+ 1037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33046: BlockingRecv error: recv got EOF from 127.0.0.1:33046 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260621 06:57:31.946486 11382 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:31.944003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.20.120.253:44405)
0621 06:57:31.944217 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.944225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.944834 (+ 609us) server_negotiation.cc:407] Connection header received
0621 06:57:31.945049 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.945055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.945161 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.945283 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.946207 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33058: BlockingRecv error: recv got EOF from 127.0.0.1:33058 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260621 06:57:31.984243 11382 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:31.981942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.20.120.253:44405)
0621 06:57:31.982230 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:31.982241 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:31.982869 (+ 628us) server_negotiation.cc:407] Connection header received
0621 06:57:31.983061 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:31.983066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:31.983160 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:31.983288 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:31.983984 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33070: BlockingRecv error: recv got EOF from 127.0.0.1:33070 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260621 06:57:32.053340 11382 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:32.051651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.20.120.253:44405)
0621 06:57:32.051839 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:32.051844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:32.052143 (+ 299us) server_negotiation.cc:407] Connection header received
0621 06:57:32.052313 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:32.052319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:32.052401 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:32.052496 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:32.053174 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33084: BlockingRecv error: recv got EOF from 127.0.0.1:33084 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260621 06:57:32.186867 11382 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:32.184902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.20.120.253:44405)
0621 06:57:32.185168 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:32.185175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:32.185443 (+ 268us) server_negotiation.cc:407] Connection header received
0621 06:57:32.185630 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:32.185634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:32.185707 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:32.185793 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:32.186646 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33092: BlockingRecv error: recv got EOF from 127.0.0.1:33092 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
I20260621 06:57:32.207644 11470 catalog_manager.cc:1259] Loaded cluster ID: 970f9673d4ca4d149dd80b6dffbda833
I20260621 06:57:32.207734 11470 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e: loading cluster ID for follower catalog manager: success
I20260621 06:57:32.210114 11470 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e: acquiring CA information for follower catalog manager: success
I20260621 06:57:32.211730 11470 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ce3985d509c407792d39b7cbf5f952e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 06:57:32.448221 11382 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 06:57:32.446528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.20.120.253:44405)
0621 06:57:32.446812 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:32.446820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:32.446958 (+ 138us) server_negotiation.cc:407] Connection header received
0621 06:57:32.447156 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:32.447160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:32.447229 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:32.447315 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 06:57:32.447985 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33096: BlockingRecv error: recv got EOF from 127.0.0.1:33096 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
I20260621 06:57:32.492461 11518 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38: acquiring CA information for follower catalog manager: success
I20260621 06:57:32.494481 11518 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac8f0dc3e1a144d09e0597a1eb991b38: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 06:57:32.969362 11584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:32.963429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.20.120.253:44405)
0621 06:57:32.963910 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:32.963916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:32.964038 (+ 122us) server_negotiation.cc:407] Connection header received
0621 06:57:32.964282 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:32.964288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:32.964456 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:32.964599 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:57:32.965049 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:32.965751 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:32.967103 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:32.967580 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:32.967711 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:32.968189 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:57:32.968769 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:57:32.968941 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:32.968996 (+ 55us) server_negotiation.cc:299] Negotiation successful
0621 06:57:32.969076 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
I20260621 06:57:32.980173 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:32.974849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40992 (local address 127.20.120.252:33597)
0621 06:57:32.975315 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:32.975320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:32.975337 (+ 17us) server_negotiation.cc:407] Connection header received
0621 06:57:32.975515 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:32.975519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:32.975661 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:32.975791 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:57:32.976182 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:32.976854 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:32.978133 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:32.978530 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:32.978621 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:32.979105 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:57:32.979672 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:57:32.979778 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:32.979895 (+ 117us) server_negotiation.cc:299] Negotiation successful
0621 06:57:32.979956 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":258,"threads_started":1}
I20260621 06:57:32.993789 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:32.985998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.20.120.252:33597)
0621 06:57:32.986152 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:32.986156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:32.987691 (+ 1535us) server_negotiation.cc:407] Connection header received
0621 06:57:32.987756 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:32.987761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:32.987848 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:32.987957 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:57:32.988824 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:32.989774 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:32.991639 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:32.992218 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:32.992339 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:32.992812 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:57:32.992929 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:57:32.993033 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:32.993415 (+ 382us) server_negotiation.cc:299] Negotiation successful
0621 06:57:32.993481 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 06:57:32.993783 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:32.985870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.20.120.254:37429)
0621 06:57:32.986441 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:32.986449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:32.986772 (+ 323us) server_negotiation.cc:407] Connection header received
0621 06:57:32.987056 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:32.987082 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:32.987316 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:32.987510 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:57:32.988373 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:32.989456 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:32.991363 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:32.991919 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:32.992042 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:32.992618 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:57:32.993058 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:57:32.993190 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:32.993389 (+ 199us) server_negotiation.cc:299] Negotiation successful
0621 06:57:32.993473 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":276,"threads_started":1}
I20260621 06:57:32.994616 11584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 06:57:32.987527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.20.120.253:44405)
0621 06:57:32.987735 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 06:57:32.987743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 06:57:32.988352 (+ 609us) server_negotiation.cc:407] Connection header received
0621 06:57:32.988645 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 06:57:32.988650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 06:57:32.988753 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 06:57:32.988835 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 06:57:32.989417 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:32.990571 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:32.992477 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 06:57:32.993012 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 06:57:32.993158 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 06:57:32.993845 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 06:57:32.994024 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 06:57:32.994105 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 06:57:32.994282 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 06:57:32.994354 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 06:57:32.999585 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 11237
I20260621 06:57:33.000646 11299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:57:33.110476 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 11237
I20260621 06:57:33.128521 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 11311
I20260621 06:57:33.129594 11373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:57:33.236851 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 11311
I20260621 06:57:33.257342 20963 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu pid 11385
I20260621 06:57:33.258445 11447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 06:57:33.362047 20963 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcf0MiJ/build/asan/bin/kudu with pid 11385
2026-06-21T06:57:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4994 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9922 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (352692 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST